Powered By Blogger

Tuesday, July 2, 2019

Kafka Producer in main program


package com.tgt.hdfsToKafka;

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Properties;
import java.util.concurrent.ExecutionException;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class TestProducer {

public static void main(String[] args) throws IOException {
try {
Producer producer = createProducer();
String message = readFile();

ProducerRecord record = new ProducerRecord<>("scm-testing",
"testmessagekey", message);

producer.send(record).get();
} catch (InterruptedException | ExecutionException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

private static Producer createProducer() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
"broker:9093");
props.put(ProducerConfig.CLIENT_ID_CONFIG, "hdfsToKafka");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// props.put("ssl.truststore.location",
// "/Users/basan/Documents/JavaProjects/hdfsToKafka/src/main/resources/client.truststore.jks");
props.put("ssl.truststore.location", "/Users/basan/Desktop/asfasf/kafka_cert/dddddddclient.truststore.jks");
props.put("ssl.truststore.password", "jansfjjasjfjasjfjj);
// props.put("ssl.keystore.location",
// "/Users/basan/Documents/JavaProjects/hdfsToKafka/src/main/resources/ffods-stg.target.com.jks");
props.put("ssl.keystore.location", "/Users/basan/Desktop/asfasf/kafka_cert/serversfasfasfasfasf.keystore.jks");
props.put("ssl.keystore.password", "nasf as f n n n");
props.put("security.protocol", "SSL");
props.put("ssl.protocol", "TLSv1.2");
return new KafkaProducer<>(props);
}
static String readFile() throws IOException{
String contents = new String(Files.readAllBytes(Paths.get("/Users/basan/Documents/workspace-sts-3.8.4.RELEASE/hdfsToKafka/sample.json")));

return contents;
/*InputStream is = new FileInputStream("/Users/basan/Documents/workspace-sts-3.8.4.RELEASE/hdfsToKafka/sample.json");
BufferedReader buf = new BufferedReader(new InputStreamReader(is));
       
String line = buf.readLine();
StringBuilder sb = new StringBuilder();
       
while(line != null){
  sb.append(line);
  line = buf.readLine();
}
       
String fileAsString = sb.toString();
System.out.println("Contents : " + fileAsString);
return fileAsString;*/
}

}

Monday, July 1, 2019

alfresco smart folder

Below are the steps
  1. upload the attached json file having smart folder definition to smart folder template and change type to smart folder template. 
  2. Create the site and under documentLibrary create folder
  3. Create contents by applying the aspect 
  4. On the folder where we would like to see the smart folder apply aspect System smart folder
  5. Click on edit properties and select template json file uploaded
  6. We should be able to see below structure.

Wednesday, April 24, 2019

MapReduce materials


This has good explanation for java MR

This has good info on MR related to combiner,partitioner


 Mapper

Keyin, valuein, keyoutm , valueoutm

Reducer

Keyinm, valueinm, keyout , valueout


keyoutm  will be passed to reducer as Keyinm
Valueoutm will be passed to reducer as valueinm


MRUnit can be used for doing MR testing.







mapreduce-job-execution-flow-1-1024x492-1.jpg (1200×628)






Wednesday, April 3, 2019

Chef

Chef is configuration management tool for the hardware. It will be used to install softwarres, create folders with permissions, for doing patching and to open the ports.


Chef will have the workstation where the cookbook is developed having details about what to do when it gets executed on the node.


knife is the utility used to talk with chef server. It can be used to see the existing cookbooks, pushing cookbook to server, pushing cookbook to node etc
Knife needs to be configured with the chef server details.

Chef Server : it stores cookbooks , roles and users

Node : The node where the cookbook instructions will be executed.
sudo chef-client
is the command to be used to get the latest cookbooks from server.
Node will have the chef agent installed which will keep on talking to server.


https://www.youtube.com/watch?v=2BBC6sb4FQ0

https://www.youtube.com/watch?v=jlwGcgFfcnU

Monday, March 11, 2019

Kafka Utils

Getting message from speific offset
./bin/kafka-console-consumer.sh --bootstrap-server kafka-broker.service-pe-test.consul.c-test.ost.cloud.internal:9093 --max-messages 1 --group basan_PERF_console --consumer.config config-stage.properties --partition 1 --offset 14677669 --topic order-demand



config-stage.properties
ssl.truststore.location=/Users/basan/Documents/workspace-sts-3.8.4.RELEASE/ds-conumption-spring/ssl-config/prod/client.truststore.jks
ssl.truststore.password=**********
ssl.keystore.location=/Users/basan/Documents/workspace-sts-3.8.4.RELEASE/ds-conumption-spring/ssl-config/stg/ds-stg.target.com.jks
ssl.keystore.password=**********
security.protocol=SSL
ssl.protocol=TLSv1.2



Get lag for a specific consumner group

./bin/kafka-consumer-groups.sh --bootstrap-server kafka-broker.service-pe-test.consul.c-test.ost.cloud.internal:9093 --group OPROD12CV1_SHIPMENT_HEADER --describe --command-config config-stage.properties


Wednesday, February 20, 2019

Sqoop import to DB and export to DB

For importing from database to Hive

sqoop import -D mapreduce.job.queuename=abcdef --connect jdbc:oracle:thin:@dbdddd-sc8:1521/dbservice1 --username userrrrr --password cdwewerewrewrwerV   --split-by 'modifyts'  --hive-import  --hive-table 'abd.det_landing'  --delete-target-dir --target-dir 'hdfs://bigredns/apps/hive/warehouse/f abd.db/det_landing' --query " select header_key ,  document_type ,  order_date ,  modifyts ,  status ,  order_no ,  shipping_service ,  item_id ,  item_short_description ,  fulfillment_type ,  prime_line_no ,  extn_guest_sel_ship_opt ,  unit_price ,  line_total ,  status_date ,  status_quantity ,  scac ,  carrier_service_code ,  aged_order ,  req_ship_date ,  must_ship_date ,  ship_advice_no ,  shipnode_type ,  shipnode_name ,  node_description ,  status ,  status_desc ,  expected_date ,  actual_date , city ,line_key ,  order_release_status_key ,  reason_code,wcs_guest_id,btch_ld_d,two_days_shipping_f from mgr.ord_det_vb12  where trunc(btch_ld_d)=trunc(sysdate -1) and   \$CONDITIONS"    -m 30 --fetch-size 10000   --fields-terminated-by '\001'  --lines-terminated-by '\n'



Export to Oracle db from the filesystem

sqoop export -D mapreduce.job.queuename= abcdef -D mapreduce.map.memory.mb=2048 \
--connect jdbc:oracle:thin:@db-sc8:1521/dbservice1 \
--username user \
--password jjnnsnnnnpassword \
--num-mappers 10 \
--input-fields-terminated-by ";" \
--table MGR.ord_MTH \
--export-dir /common/abcd/def_Output/method*