cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Data_Engineer3
by Contributor II
  • 5111 Views
  • 4 replies
  • 4 kudos

Resolved! Unable to read file from dbfs location in databricks.

When i tried to read file from dbfs, it throws error - Caused by: FileReadException: Error while reading file dbfs:/.......................parquet is not a Parquet file. Expected magic number at tail [80, 65, 82, 49] but found [105, 108, 101, 115].Bu...

  • 5111 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @KARTHICK N​, What's the one-line code you're trying to read the file, precisely the path?Can you confirm if your file is a CSV or Parquet file?Are you trying to read it in python or scala?

  • 4 kudos
3 More Replies
AJ270990
by Contributor II
  • 13680 Views
  • 3 replies
  • 0 kudos

Resolved! I am getting ParseException: error while running the spark SQL query

I am using below code to create the Spark session and also loading the csv file. Spark session and loading csv is running well. However SQL query is generating the Parse Exception.%pythonfrom pyspark.sql import SparkSession     # Create a SparkSessio...

  • 13680 Views
  • 3 replies
  • 0 kudos
Latest Reply
AJ270990
Contributor II
  • 0 kudos

This is resolved. Below query works fine nowsqldf = spark.sql("select sum(cast(enrollment as float)), sum(cast(growth as float)),`plan type`,`Parent Organization`,state,`Special Needs Plan`,`Plan Name Sec A`, CASE when `Plan ID` between '800' and '89...

  • 0 kudos
2 More Replies
Jhaji
by New Contributor
  • 560 Views
  • 0 replies
  • 0 kudos

The REFRESH TABLE command doesn't seem to invalidate the local cache. Am I missing something?

Hi Team,As part of "Data Enginering with Databricks" course section "DE 4.2 - Providing Options for External Sources", I can read total number of records of sales_csv table as 10510. The append command in Cmd17 is supposed to increase this number 2x,...

image
  • 560 Views
  • 0 replies
  • 0 kudos
159312
by New Contributor III
  • 1538 Views
  • 3 replies
  • 0 kudos

When trying to ingest parquet files with autoloader I get an error stating that schema inference is not supported, but the parquet files have schema data. No inference should be necessary. Is this right?

When trying to ingest parquet files with autoloader with the following codedf = (spark   .readStream   .format("cloudFiles")   .option("cloudfiles.format","parquet")   .load(filePath))I get the following error:java.lang.UnsupportedOperationException:...

  • 1538 Views
  • 3 replies
  • 0 kudos
Latest Reply
Noopur_Nigam
Valued Contributor II
  • 0 kudos

Hi @Ben Bogart​ This is supported in DBR 11.1 and above.The below document suggests the same:https://docs.databricks.com/ingestion/auto-loader/schema.html#schema-inference-and-evolution-in-auto-loaderPlease try in DBR 11.1 and please let us know if y...

  • 0 kudos
2 More Replies
Zair
by New Contributor III
  • 1288 Views
  • 2 replies
  • 2 kudos

How to handle 100+ tables ETL through spark structured streaming?

I am writing a streaming job which will be performing ETL for more than 130 tables. I would like to know is there any other better way to do this. Another solution I am thinking is to write separate streaming job for all tables. source data is coming...

  • 1288 Views
  • 2 replies
  • 2 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 2 kudos

Hi, I guess to answer your question it might be helpful to get more details on what you're trying to achieve and the bottleneck that you encounter now.Indeed handle the processing of 130 tables in one monolith could be challenging as the business rul...

  • 2 kudos
1 More Replies
AJ270990
by Contributor II
  • 6657 Views
  • 3 replies
  • 4 kudos

Resolved! How to bold a text ?

I have searched several ways on applying a bold to a text however unable to achieve it.Have added '\033[1m' then my text and followed by '\033[0m', however cant see the text as bold.I need to apply Bold to the Header "Ocean" in below image which is i...

image
  • 6657 Views
  • 3 replies
  • 4 kudos
Latest Reply
AJ270990
Contributor II
  • 4 kudos

I have used plt.text() to make text bold

  • 4 kudos
2 More Replies
RaymondLC92
by New Contributor II
  • 1303 Views
  • 2 replies
  • 1 kudos

Resolved! How to obtain run_id without using dbutils in python?

We would like to be able to get the run_id in a job run and we have the unfortunate restriction that we cannot use dbutils, is there a way to get it in python?I know for Job ID it's possible to retrieve it from the environment variables.

  • 1303 Views
  • 2 replies
  • 1 kudos
Latest Reply
artsheiko
Valued Contributor III
  • 1 kudos

Hi, please refer to the following thread : https://community.databricks.com/s/question/0D58Y00008pbkj9SAA/how-to-get-the-job-id-and-run-id-and-save-into-a-databaseHope this helps

  • 1 kudos
1 More Replies
Rahul_Samant
by Contributor
  • 5362 Views
  • 4 replies
  • 5 kudos

Resolved! High Concurrency Pass Through Cluster : pyarrow optimization not working while converting to pandasdf

i need to convert a spark dataframe to pandas dataframe with arrow optimization spark.conf.set("spark.sql.execution.arrow.enabled", "true")data_df=df.toPandas()but getting one of the below error randomly while doing so Exception: arrow is not support...

  • 5362 Views
  • 4 replies
  • 5 kudos
Latest Reply
AlexanderBij
New Contributor II
  • 5 kudos

Can you confirm this is a known issue?Running into same issue, example to test in 1 cell.# using Arrow fails on HighConcurrency-cluster with PassThrough in runtime 10.4 (and 10.5 and 11.0)   spark.conf.set("spark.sql.execution.arrow.pyspark.enabled",...

  • 5 kudos
3 More Replies
NickMendes
by New Contributor III
  • 1680 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks SQL duplicates alert e-mail

Hi everyone, I've been working in DBC SQL and creating few e-mails alerts and have noticed that when it triggers, e-mail notification is getting duplicated. I've been trying lots of testing in different situations, however, it keeps duplicating in my...

  • 1680 Views
  • 3 replies
  • 1 kudos
Latest Reply
NickMendes
New Contributor III
  • 1 kudos

After lots of testing, I've finally figured out one solution. I've changed "Notifications" settings to "When triggered, send notification At most every 1 day" and "Refresh" to "Refresh every 1 day". Now it is working perfectly.

  • 1 kudos
2 More Replies
harsha4u
by New Contributor II
  • 677 Views
  • 1 replies
  • 2 kudos

Any suggestions around automating sizing of clusters and best practices around it? Other than enabling auto scaling, are there any other practices aro...

Any suggestions around automating sizing of clusters and best practices around it? Other than enabling auto scaling, are there any other practices around creating a right size driver/worker nodes?

  • 677 Views
  • 1 replies
  • 2 kudos
Latest Reply
User16766737456
New Contributor III
  • 2 kudos

Autoscaling should help in sizing the clusters according to the workload. You may want to consider the recommendations here: https://docs.databricks.com/clusters/cluster-config-best-practices.html#cluster-sizing-considerations

  • 2 kudos
Valentin1
by New Contributor III
  • 1194 Views
  • 3 replies
  • 0 kudos

docs.databricks.com

Is there an any example that contains at least one widget that works with the Databricks SQL Create Dashboard API? I tried the following simple dashboard:{ "name": "Delta View", "dashboard_filters_enabled": false, "widgets": [ { ...

  • 1194 Views
  • 3 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

@Valentin Rosca​ , Right now, Databricks also does not recommend creating new widgets via queries and dashboards API (https://docs.databricks.com/sql/api/queries-dashboards.html#operation/sql-analytics-create-dashboard).  Also, copying a dashboard fr...

  • 0 kudos
2 More Replies
Isaac_Low
by New Contributor II
  • 1228 Views
  • 3 replies
  • 3 kudos
  • 1228 Views
  • 3 replies
  • 3 kudos
Latest Reply
Isaac_Low
New Contributor II
  • 3 kudos

All good. I just imported the training material manually using the dbc link. Didn't need repos for that.

  • 3 kudos
2 More Replies
davidvb
by New Contributor II
  • 1663 Views
  • 2 replies
  • 1 kudos

I have a big problem creating a community account

It is impossible for me create a community account. I put my data on web and in the next step, when the website show me the 3 type of data ( google, amazn etc) and I click on the “ "Get started with community account" the web show me this  I have try...

problem
  • 1663 Views
  • 2 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Moderator
  • 1 kudos

Hi @david vazquez​,It seems like the website was down due to maintenance. You can check the status page next time to check why the website is down https://status.databricks.com/

  • 1 kudos
1 More Replies
darshan
by New Contributor III
  • 1114 Views
  • 3 replies
  • 1 kudos

job init takes longer than notebook run

I am trying to understand why running a job takes longer than running the notebook manually.And if I try to run jobs concurrently using workflow or threads then is there a way to reduce job init time ?

  • 1114 Views
  • 3 replies
  • 1 kudos
Latest Reply
Vivian_Wilfred
Honored Contributor
  • 1 kudos

Hi @darshan doshi​ , Jobs creates a job cluster in the backend before it starts the task execution and this cluster creation may take extra time when compared to running a notebook on a existing cluster.1) If you run a multi-task job, you could selec...

  • 1 kudos
2 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels