cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

rt-slowth
by Contributor
  • 770 Views
  • 0 replies
  • 0 kudos

Notebook of Databricks's result

If there is no data abnormality in redshift connecting to spark from shared in databricks, and the data suddenly decreases, what cause should I check? Also, is there any way to check the variables in widget or code on each execution?

  • 770 Views
  • 0 replies
  • 0 kudos
RantoB
by Valued Contributor
  • 21296 Views
  • 7 replies
  • 4 kudos

Resolved! read csv directly from url with pyspark

I would like to load a csv file directly to a spark dataframe in Databricks. I tried the following code :url = "https://opendata.reseaux-energies.fr/explore/dataset/eco2mix-national-tr/download/?format=csv&timezone=Europe/Berlin&lang=fr&use_labels_fo...

  • 21296 Views
  • 7 replies
  • 4 kudos
Latest Reply
MartinIsti
New Contributor III
  • 4 kudos

I know it's a 2 years old thread but I needed to find a solution to this very thing today. I had one notebook using SparkContextfrom pyspark import SparkFilesfrom pyspark.sql.functions import *sc.addFile(url) But according to the runtime 14 release n...

  • 4 kudos
6 More Replies
vlado101
by New Contributor II
  • 3488 Views
  • 1 replies
  • 1 kudos

Resolved! ANALYZE TABLE is not updating columns stats

Hello everyone,So I am having an issue when running "ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS". The way I understand it this should update the min/max value for a column when you run it for all or one column. One way to verify it from what I ...

  • 3488 Views
  • 1 replies
  • 1 kudos
Latest Reply
Priyanka_Biswas
Databricks Employee
  • 1 kudos

Hello @vlado101  The ANALYZE TABLE COMPUTE STATISTICS FOR ALL COLUMNS command in Databricks is used to compute statistics for all columns of a table. This information is persisted in the metastore and helps the query optimizer make decisions such as ...

  • 1 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 1428 Views
  • 1 replies
  • 1 kudos

Structured Streaming Aggregation

Utilizing structured streaming to read the change data feed from your Delta table empowers you to execute incremental streaming aggregations, such as counting and summing.

structured2.png
  • 1428 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Thank you for sharing @Hubert-Dudek !!!

  • 1 kudos
Kayla
by Valued Contributor
  • 3778 Views
  • 2 replies
  • 2 kudos

Resolved! Paramiko SFTP Get fails on databricks file system

I have an SFTP server I need to routinely download Excel files from and put into GCP cloud storage buckets.Every variation of the filepath to either my GCP path or just the dbfs in-built file system is giving an error of " [Errno 2] No such file or d...

  • 3778 Views
  • 2 replies
  • 2 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 2 kudos

Thank you for sharing the solution. Many more users will find this information very useful. 

  • 2 kudos
1 More Replies
Gobo
by New Contributor II
  • 1602 Views
  • 1 replies
  • 0 kudos

Connecting Power BI to Delta Share: Error on data retrieval

Hi everybody,sharing data with an access token and Databricks Connector works fine in Power BI (desktop). Now we wanted to switch to Delta Sharing.We setup a delta share to distribute data via open share to anyone outside our organization. Unity Cata...

pb_error.jpg
Data Engineering
delta share
Power BI
  • 1602 Views
  • 1 replies
  • 0 kudos
Latest Reply
Gobo
New Contributor II
  • 0 kudos

Hi everybody,for anybody running into the same issue.It is a bug in the current Power Bi version (2.121.644.0). I reverted back to the April release (2.116.404.0), which does work as expected. 

  • 0 kudos
User16790091296
by Contributor II
  • 3027 Views
  • 1 replies
  • 0 kudos

How to create a databricks job with parameters via CLI?

I'm creating a new job in databricks using the databricks-cli:databricks jobs create --json-file ./deploy/databricks/config/job.config.jsonWith the following json:{ "name": "Job Name", "new_cluster": { "spark_version": "4.1.x-scala2.1...

  • 3027 Views
  • 1 replies
  • 0 kudos
Latest Reply
matthew_m
Databricks Employee
  • 0 kudos

This is an old post but still relevant for future readers, so will answer how it is done. You need to add base_parameters flag in the notebook_task config, like the following.   "notebook_task": { "notebook_path": "...", "base_parameters": { ...

  • 0 kudos
samst
by New Contributor III
  • 5827 Views
  • 9 replies
  • 6 kudos

Resolved! Spark UI reverse Proxy blocked on GCP

Using the 9.1ML cluster atm but also tried the 7.3 and 8.1.Databricks is deployed on google platform and I was using the trial.It is quite difficult to debug if the spark ui is only semi accessible.Part of the results in raw html are visible but all ...

  • 5827 Views
  • 9 replies
  • 6 kudos
Latest Reply
LucasArrudaW
New Contributor II
  • 6 kudos

Any news about this?

  • 6 kudos
8 More Replies
lndlzy
by New Contributor II
  • 4392 Views
  • 2 replies
  • 0 kudos

ADD_NODES_FAILED Cluster Does Not Start

Hello everyone, I tried to change a Databricks Runtime Cluster from 12.2 LTS ML to 13.3 LTS ML, however I got this error: Failed to add 1 container to the compute. Will attempt retry: false. Reason: Global init script failureGlobal init script Instal...

  • 4392 Views
  • 2 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
Hello everyone, I tried to change a Databricks Runtime Cluster from 12.2 LTS ML to 13.3 LTS ML, however I got this error: Failed to add 1 container to the compute. Will attempt retry: false. Reason: Global init script failureGlobal init script Instal...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.
1 More Replies
TimReddick
by Contributor
  • 7622 Views
  • 6 replies
  • 2 kudos

Using run_job_task in Databricks Asset Bundles

Do Databrick Asset Bundles support run_job_task tasks?I've made various attempts to add a run_job_task with a specified job_id. See my the code_snippet below. I tried substituting the job_id using ${...} syntax, as well as three other ways which I've...

Data Engineering
Databrick Asset Bundles
run_job_task
  • 7622 Views
  • 6 replies
  • 2 kudos
Latest Reply
kyle_r
New Contributor II
  • 2 kudos

Ah, I see it is a known bug in the Databricks CLI: Asset bundle run_job_task fails · Issue #812 · databricks/cli (github.com). Anyone facing this issue should comment on and keep an eye on that ticket for resolution. 

  • 2 kudos
5 More Replies
User16765131552
by Contributor III
  • 6062 Views
  • 3 replies
  • 0 kudos

Resolved! Pull Cluster Tags

Does anybody know any in-notebook or JAR code to pull cluster tags from the runtime environment? Something like... dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().apply('user')but for the cluster name?

  • 6062 Views
  • 3 replies
  • 0 kudos
Latest Reply
DatBoi
Contributor
  • 0 kudos

Did you find any documentation for spark.conf.get properties? I am trying to get some metadata about the environment my notebook is running in (specifically cluster custom tags)? But cannot find any information beside a couple of forum posts.

  • 0 kudos
2 More Replies
arielmoraes
by New Contributor III
  • 2556 Views
  • 1 replies
  • 1 kudos

Resolved! Job Concurrency Queue not working as expected

I have a process that should run the same notebook with varying parameters, thus translating to a job with queue and concurrency enabled. When the first executions are triggered the Jobs Runs work as expected, i.e. if the job has a max concurrency se...

arielmoraes_0-1696872175101.png arielmoraes_1-1696872724206.png
  • 2556 Views
  • 1 replies
  • 1 kudos
Latest Reply
arielmoraes
New Contributor III
  • 1 kudos

Hi @Retired_mod, we double-checked everything, the resources are enough and all settings are properly set. I'll reach out the support by filing a new ticket. Thank you for your help.

  • 1 kudos
b_1
by New Contributor II
  • 1214 Views
  • 2 replies
  • 1 kudos

to_timstamp function in non-legacy mode does not parse this format: yyyyMMddHHmmssSS

I have this datetime string in my dataset: '2023061218154258' and I want to convert it to datetime, using below code. However the format that I expect to work, doesn't work, namely: yyyyMMddHHmmssSS. This code will reproduce the issue:from pyspark.sq...

  • 1214 Views
  • 2 replies
  • 1 kudos
Latest Reply
b_1
New Contributor II
  • 1 kudos

Is there anybody who has the same issue or knows that this is in fact an issue?

  • 1 kudos
1 More Replies
orso
by New Contributor III
  • 8933 Views
  • 1 replies
  • 0 kudos

Resolved! Java - FAILED_WITH_ERROR when saving to snowflake

I'm trying to move data from database A to B on Snowflake. There's no permission issue since using the Python package snowflake.connector  works Databricks runtime version: 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12)Insert into database B fail...

  • 8933 Views
  • 1 replies
  • 0 kudos
Latest Reply
orso
New Contributor III
  • 0 kudos

Found the problem. The sub-roles didn't have grants to the warehouse.I hope it will help someone one day

  • 0 kudos
sanjay
by Valued Contributor II
  • 2245 Views
  • 1 replies
  • 0 kudos

Trigger Events in data pipeline

Hi,I am running datapipeline in databrick using matillion architecture. I am facing inconsistent events in silver to gold layer in case any row deleted/updated from a partition. Let me explain with example.e.g. I have data in silver layer with partit...

  • 2245 Views
  • 1 replies
  • 0 kudos
Latest Reply
sanjay
Valued Contributor II
  • 0 kudos

Thank you Kaniz. Further queries on this.1. If I have nested partitions e.g. on department & date, finance->09, finance->10 and if am updating one record in finance->09 then will then updates partition finance->10 as well2. Is it good idea to have sm...

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels