cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
vivek_sinha
by • Contributor
  • 7369 Views
  • 3 replies
  • 4 kudos

Resolved! Getting Authentication Error while accessing Azure Blob table (wasb) URL using PySpark

I am trying to access the Azure Blob table using Pyspark but getting an Authentication Error. Here I am passing SAS token (HTTP and HTTPS enabled) but it's working only with WASBS (HTTPS) URL, not with WASB (HTTP) URL.Even I tried with Account key as...

Data Engineering
Reply
  • 7369 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
vivek_sinha
Contributor
  • 4 kudos

Hi @Arvind Ravish​  The issue got fixed after passing HTTP and HTTPS enabled token to spark executors.Thanks again for your help

  • 4 kudos
2 More Replies
Prabakar
by • Esteemed Contributor III
  • 2330 Views
  • 1 replies
  • 1 kudos

Non-admin users unable to create jobs from Job UI Non-admin users may be experiencing difficulties interacting with the jobs UI. This is due to a rece...

Non-admin users unable to create jobs from Job UINon-admin users may be experiencing difficulties interacting with the jobs UI. This is due to a recently discovered UI regression in the 3.73 shard release, deployed to the jobs service starting June 6...

Data Engineering
Reply
  • 2330 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 1 kudos

This has been conveyed to all customers. If the email landed in your spam box then this should help you.

  • 1 kudos
mali_bigdata
by • New Contributor
  • 1030 Views
  • 0 replies
  • 0 kudos

Databricks is adding NULL value in the URL while moving the Fairlearn dashboard and causing CORS error and fairlearn dashboard keeps spinning.

We are trying to run FairnessDashboard and once we pass in the data to the dashboard it keeps on the spinning. Please see the attached file.Also we noticed that Databricks is adding NULL in the URL and eventually we get the CORS error and it is redir...

Data Engineering
Reply
  • 1030 Views
  • 0 replies
  • 0 kudos
eager_to_learn
by • New Contributor III
  • 4462 Views
  • 7 replies
  • 5 kudos

Resolved! Databricks pool - 2 instances are in running state without any job running in the system

We are using Azure Databricks pools, configured 16 max instances. Out of 16, 2 instances are in running state without any job in running condition, how & where can i check the usage of the instances ?p.s. SQL pool is also not running, so no chances o...

Data Engineering
Reply
  • 4462 Views
  • 7 replies
  • 5 kudos
View Replies
Latest Reply
eager_to_learn
New Contributor III
  • 5 kudos

@Kaniz Fatma​ / @Prabakar Ammeappin​ Any idea, how can we queue the jobs in the Resource pools, is it some setting which we need to switch on so the jobs are queued until instances are available or can you point some documentation for the same ?

  • 5 kudos
6 More Replies
ABAGRI
by • New Contributor II
  • 1743 Views
  • 2 replies
  • 2 kudos

Resolved! Having Issues with extracting records from complex JSON

Hi Team,we are using delta live tables to ingest data from Kafka.the JSON file we receive is a complex JSON structure and we are trying to explode the file into its necessary columns and transactions, Thank youplease see attached sample file{ "Table...

Data Engineering
Reply
  • 1743 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
User16753725469
Contributor II
  • 2 kudos

Hi @Lantis Pillay​ Could you please try to parse JSON records in the below way

  • 2 kudos
1 More Replies
MattM
by • New Contributor III
  • 2058 Views
  • 0 replies
  • 0 kudos

Unstructured Data - PDF and a semi-structured data

I have a scenario where one source is unstructered pdf files and another source is semi-structered JSON files. I get files from these two sources on a daily basis into an ADLS storage. What is the best way to load this into a medallion structure by s...

Data Engineering
Reply
  • 2058 Views
  • 0 replies
  • 0 kudos
Antoine_De_A
by • New Contributor III
  • 2507 Views
  • 1 replies
  • 3 kudos

Resolved! Streaming data to CosmosDB

Hello everyone,Here is the problem I am facing. I'm currently working on streaming data to DataBricks, my goal is to create a data stream on a first notebook, and then on a second notebook to read this data stream, add all the new rows to a dataFrame...

Data Engineering
Reply
  • 2507 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Antoine_De_A
New Contributor III
  • 3 kudos

Problem solved!Instead of trying to do everything directly with the .writeStream options I used the .forEachBatch() function which allows me to call a function outside the .writeStream().In this function I get a dataFrame in parameter which is my str...

  • 3 kudos
curious-case-of
by • New Contributor II
  • 9593 Views
  • 1 replies
  • 4 kudos

Databricks notebook taking too long to run as a job compared to when triggered from within the notebook

I don't know if this question has been covered earlier, but here it goes - I have a notebook that I can run manually using the 'Run' button in the notebook or as a job.The runtime when I run from within the notebook directly is roughly 2 hours. But w...

Data Engineering
Reply
  • 9593 Views
  • 1 replies
  • 4 kudos
View Replies
Latest Reply
wvl
New Contributor II
  • 4 kudos

We're seeing the same behavior.. Good performance using interactive cluster.Using identically sized job cluster, performance is bad. Any ideas?

  • 4 kudos
data_engineer_0
by • New Contributor II
  • 13714 Views
  • 3 replies
  • 2 kudos

How to run the .py file in databricks cluster

Hi team,I wants to run the below command in databricks and also need to capture the error and success message.Please help me out here,Thanks in advanceEx: python3 /mnt/users/code/x.py --arguments

Data Engineering
Reply
  • 13714 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
User16764241763
Honored Contributor
  • 2 kudos

Hello @Piper Wilson​ Would this task not help?https://docs.databricks.com/dev-tools/api/latest/examples.html#jobs-api-examples

  • 2 kudos
2 More Replies
User15787040559
by • New Contributor III
  • 2497 Views
  • 1 replies
  • 0 kudos

MicrosoftTeams-image

ERROR Max retries exceeded with url: /api/2.0/jobs/runs/get?run_id= Failed to establish a new connectionThis error can happen when exceeding the rate limits for all REST API calls as documented here.In the image shown for example we're using the Jobs...

Data Engineering
Reply
  • 2497 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16764241763
Honored Contributor
  • 0 kudos

Hi @Carlos Morillo​  Are you facing this issue consistently or when you run a lot of jobs?We are internally tracking a similar issue. Could you please file a support request with Microsoft Support? Databricks and MSFT will collaborate and provide upd...

  • 0 kudos
chandan_a_v
by • Valued Contributor
  • 18594 Views
  • 7 replies
  • 3 kudos

Resolved! TypeError: 'JavaPackage' object is not callable

image
Data Engineering
Reply
  • 18594 Views
  • 7 replies
  • 3 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 3 kudos

By any chance, was the cluster restarted after installing the libraries or was it detached and reattached from/to the notebook? Notebook-scoped libraries do not persist across sessions. You must reinstall notebook-scoped libraries at the beginning of...

  • 3 kudos
6 More Replies
Gopal_Sir
by • New Contributor III
  • 28377 Views
  • 5 replies
  • 7 kudos

Resolved! How to convert a string column to Array of Struct ?

I have a nested struct , where on of the field is a string , it looks something like this ....string = "[{\"to_loc\":\"6183\",\"to_loc_type\":\"S\",\"qty_allocated\":\"18\"},{\"to_loc\":\"6137\",\"to_loc_type\":\"S\",\"qty_allocated\":\"9\"},{\"to_lo...

Data Engineering
Reply
  • 28377 Views
  • 5 replies
  • 7 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 7 kudos

Can you mark the question as answered so others can find the solution?

  • 7 kudos
4 More Replies
kerala_tourism
by • New Contributor
  • 416 Views
  • 0 replies
  • 0 kudos

Tourism attractions in kerala are described here. Kerala has a rich tourism background, which contributes much to the economy. Tourism is the way of i...

Tourism attractions in kerala are described here. Kerala has a rich tourism background, which contributes much to the economy. Tourism is the way of income for a large number of people in Kerala. National parks, wild life sanctuaries, etc. are the ma...

Data Engineering
Reply
  • 416 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels