- 1637 Views
- 1 replies
- 3 kudos
Extracting data from excel in datalake storage using openpyxl
i am trying to extract some data into databricks but tripping all over openpyxl, newish user of databricks..from openpyxl import load_workbookdirectory_id="hidden"scope="hidden"client_id="hidden"service_credential_key="hidden"container_name="hidden"s...
- 1637 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Vanessa Van Gelder​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 3 kudos
- 25176 Views
- 9 replies
- 5 kudos
Resolved! I created a data frame but was not able to see the data
Code to create a data frame:from pyspark.sql import SparkSessionspark=SparkSession.builder.appName("oracle_queries").master("local[4]")\ .config("spark.sql.warehouse.dir", "C:\\softwares\\git\\pyspark\\hive").getOrCreate()from pyspark.sql.functions ...
- 25176 Views
- 9 replies
- 5 kudos
- 5 kudos
@ramanjaneyulu kancharla​ can you please select my answer as best answer
- 5 kudos
- 1306 Views
- 1 replies
- 0 kudos
A streaming job going into infinite looping
HiBelow i am trying to read data from kafka, determine whether its fraud or not and then i need to write it back to mongodbbelow is my code read_kafka.pyfrom pyspark.sql import SparkSession from pyspark.sql.functions import * from pyspark.sql.types i...
- 1306 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Saswata,Can you remove the filter and see if it is printing output to console?kafka_df5=kafka_df4.filter(kafka_df4.status=="FRAUD")Thanks and RegardsSwetha Nandajan
- 0 kudos
- 3689 Views
- 2 replies
- 0 kudos
Could you please guide us on connecting ServiceNow data in databricks
Would like to extract data like ticket info, resolve time, etc., from ServiceNow in databricks.Not finding much information in community and appreciate your guidance on the same.
- 3689 Views
- 2 replies
- 0 kudos
- 0 kudos
ServiceNow offers API capabilities. You can consume the ServiceNow API within a Databricks notebook to extract data from ServiceNow. Following is a suggested prompt to use with ChatGPT for example python code to connect to ServiceNow's api. PROMPT: ...
- 0 kudos
- 3901 Views
- 2 replies
- 3 kudos
Resolved! Unable to read data from ElasticSearch using Databricks (AWS) Cannot detect ES version - Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [IP:PORT]
I am trying to read data from ElasticSearch(ES Version 8.5.2) using PySpark on Databricks (13.0 (includes Apache Spark 3.4.0, Scala 2.12)). The ecosystem is on AWS.I am able to run a curl command on the Databricks notebook to the ES ip:port and fetch...
- 3901 Views
- 2 replies
- 3 kudos
- 3783 Views
- 4 replies
- 0 kudos
Resolved! Failed to merge fields 'LIFNR' and 'LIFNR'. Failed to merge incompatible data types IntegerType and StringType
I am have imported a csv file using spark.read method, i have used custom schema and declared the type of the column as string.i have delta table and the type of the column in the table is also string. I am getting failed to merge fields errors in sp...
- 3783 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi All,the issue is resolved, i have executed column conversion and from next run the code is working fine.df = spark.read.format("delta").load("/mnt/dev/deltav2/X")df= df.withColumn("LIFNR", df.LIFNR.cast("string"))df.write.format('delta').option("o...
- 0 kudos
- 1476 Views
- 2 replies
- 2 kudos
Resolved! Normalizing data from autoloader
I have data on s3 and i'm using autoloader to load the data. My json docs have fields which are array of structures. When I don't specify any schema the whole data is stored as strings even the array of structures are just a blob of string making it ...
- 1476 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Rishitha Reddy​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...
- 2 kudos
- 706 Views
- 0 replies
- 0 kudos
How to receive data from azure event hub in parquet ?
import asyncioimport osfrom azure.eventhub.aio import EventHubConsumerClientCONNECTION_STR = "Connection_string"EVENTHUB_NAME = "event_hub"async def on_event(partition_context, event): # Put your code here. # If the operation is i/o intensive, ...
- 706 Views
- 0 replies
- 0 kudos
- 6645 Views
- 0 replies
- 0 kudos
Ingesting data from oracle to databricks through IICS
While ingesting the data from oracle to databricks through IICS, target table were created however data is not getting inserted. Below is the error. Could someone please help meException occurred when initializing data session. Root cause: java.lang....
- 6645 Views
- 0 replies
- 0 kudos
- 1055 Views
- 2 replies
- 3 kudos
Getting Error when Inserting data into table with the column as bigint
Hi All,I am creating table using Databricks SQL editor. The table definition isDROP TABLE IF EXISTS [database].***_test;CREATE TABLE [database].***_jitu_test( id bigint)USING deltaLOCATION 'test/raw/***_jitu_test'TBLPROPERTIES ('delta.minReaderVersi...
- 1055 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @jitendra goswami​ We haven't heard from you since the last response from @Werner Stinckens​ r​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpf...
- 3 kudos
- 2728 Views
- 3 replies
- 3 kudos
Error while running the data engineering course notebook : "DE 2.2 - Providing Options for External Sources"
Can somebody help me fixing this problem? I am running this notebook on databricks community edition
- 2728 Views
- 3 replies
- 3 kudos
- 3 kudos
df = spark.read.format('parquet').load(path = datasource_path) df = df.select("*").toPandas() df.to_sql('users', conn, if_exists='replace', index = False)
- 3 kudos
- 1681 Views
- 3 replies
- 0 kudos
data is not written back to data lake
I have this strange case where data is not written back to data lake. I have 3 container- . Bronze, Silver and Gold. I have done the mounting and have not problem to read the source data and write it Bronze layer ( using hive meta store catalog). T...
- 1681 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Givi Salu​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
- 0 kudos
- 1621 Views
- 2 replies
- 3 kudos
loading incremental data
I want to load incremental data to the delta live table, I wrote function to load data for 10 tables, every time that I run the pipe line, some tables are empty and have a schema, and when I run again, the other tables are empty and the previous tabl...
- 1621 Views
- 2 replies
- 3 kudos
- 3 kudos
@zahra Jalilpour​ How the DLT tables and views are updated depends on the update type:Refresh all: All live tables are updated to reflect the current state of their input data sources. For all streaming tables, new rows are appended to the table.Full...
- 3 kudos
- 27574 Views
- 7 replies
- 13 kudos
Resolved! How to read data from a table into a dataframe outside of Databricks environment?
Hi, I am working on an ML project and I need to access the data in tables hosted in my Databricks cluster through a notebook that I am running locally. This has been very easy while I run the notebooks in Databricks but I cannot figure out how to do ...
- 27574 Views
- 7 replies
- 13 kudos
- 13 kudos
We can use Apis and pyodbc to achieve this. Once go through the official documentation of databricks that might be helpful to access outside of the databricks environment.
- 13 kudos
- 1280 Views
- 2 replies
- 0 kudos
DBFS option
I can't find dbfs option in my free data bricks community edition when I tried to see location of the table.
- 1280 Views
- 2 replies
- 0 kudos
- 0 kudos
This is a limitation from community edition. You can use the existing table, but when the resources get shutdown, it will flush all the saved data.
- 0 kudos