cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
amanpreetkaur
by • New Contributor III
  • 57092 Views
  • 14 replies
  • 8 kudos

How to import one databricks python notebook into another?

I have a python notebook A in Azure Databricks having import statement as below: import xyz, datetime,... I have another notebook xyz being imported in notebook A as shown in above code. When I run notebook A, it throws the following error: ImportEr...

Data Engineering
Reply
  • 57092 Views
  • 14 replies
  • 8 kudos
View Replies
Latest Reply
artsheiko
Databricks Employee
  • 8 kudos

Create a repository containing an __init__.py fileAdd your library as .py file(s). Let's imagine that our library is composed by multiple sub-folders consolidated in "my_folder", one of sub-folders is named as "math_library" and contains my_awesome_l...

  • 8 kudos
13 More Replies
techgold
by • New Contributor
  • 6383 Views
  • 9 replies
  • 2 kudos

Resolved! how to access Databricks academy with personal email

Data Engineering
Reply
  • 6383 Views
  • 9 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Amardeep Singh​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...

  • 2 kudos
8 More Replies
Mihai_Cog
by • Contributor
  • 7115 Views
  • 14 replies
  • 0 kudos

Create a Hive db in Azure Databricks with the LOCATION parameter set to an ADLS GEN2 account fails

Hello,I am trying to create a database in Azure Databricks using the abfss Location in the create database statement and it throws an exception.%sqlCREATE DATABASE IF NOT EXISTS test COMMENT "Database for Test Area" LOCATION "abfss://test@storagetemp...

Data Engineering
azure storage
Databricks
Reply
  • 7115 Views
  • 14 replies
  • 0 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

ok.Can you check this link?Because I suspect something is wrong, and so dbrx uses the account key as a default (fs.azure.account.key)

  • 0 kudos
13 More Replies
William_Scardua
by • Valued Contributor
  • 1931 Views
  • 2 replies
  • 1 kudos

regexp_extract Error when put { in pattern

HI guys,I tried to use the regex_extract but when I put { in pattern I received errordisplay( df_containerlog.withColumn('reg', regexp_extract(col('LogEntry'), "((^.*))({'data':)", 1)) ) have any ideaThank you

error-regex.png
Data Engineering
Reply
  • 1931 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Amine
Databricks Employee
  • 1 kudos

Hello William,You have to escape the { as it is a special char in case of regex."((^.*))(\{'data':)"

  • 1 kudos
1 More Replies
niklas
by • Contributor
  • 2846 Views
  • 2 replies
  • 1 kudos

Resolved! How can I specify a custom CRAN mirror to be used permanently by default when installing packages within R Notebooks?

When installing Notebook-scoped R libraries I don't want to manually specify the custom CRAN mirror each time like this:install.packages("diffdf", repos="my_custom_cran_url'')Instead I want to take the custom CRAN mirror URL by default so that I don'...

Data Engineering
Reply
  • 2846 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
niklas
Contributor
  • 1 kudos

Got solution on Stack Overflow for this problem: https://stackoverflow.com/a/76777228/18082636

  • 1 kudos
1 More Replies
Gustavo_Az
by • Contributor
  • 9936 Views
  • 3 replies
  • 3 kudos

Resolved! Error creating external location in Unity Catalog

HelloWhen I try to create an external location I get this error:Failed to access cloud storage: [AbfsRestOperationException] HTTP Error -1CustomTokenProvider getAccessToken threw com.databricks.api.base.DatabricksServiceException : INTERNAL_ERROR: Un...

Data Engineering
Reply
  • 9936 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Gustavo_Az
Contributor
  • 3 kudos

I think I sould have something missconfigured, the way I solved the problem was to re-create the workspace and start from scratch, it was a small one for testing proposes.

  • 3 kudos
2 More Replies
jaredwolf
by • New Contributor II
  • 2240 Views
  • 2 replies
  • 6 kudos

_sqldf bugs in GCP workspaces?

Utilizing GCP instances using the 12.2DBR ML runtime. Prior to ~7:10CT last night, _sqldf commands in notebooks to reference the previously executed %sql cell would work locally as well as in scheduled Workflow Job runs. Now it appears that the code ...

Data Engineering
_sqldf
GCP
spark
SparkSQL
Workflows
Reply
  • 2240 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
Kayla
Valued Contributor
  • 6 kudos

It looks like Azure was having the same issue, it might just be all 12.2 Photon clusters.https://community.databricks.com/t5/data-engineering/sqldf-bugs-in-gcp-workspaces/td-p/38578 That post says it was been fixed, but last I checked it was still fa...

  • 6 kudos
1 More Replies
Ajay-Pandey
by • Esteemed Contributor III
  • 4811 Views
  • 2 replies
  • 1 kudos

Resolved! How to solve - gRPC message exceeds maximum size 4194304 pubsub using databricks

I am getting below error while streaming the data from pubsub using databricks DLT pipelines If anyone can help to increase the gRPC message size will help alot. 

ggg.jpg
Data Engineering
pubsub
Reply
  • 4811 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 1 kudos

   

  • 1 kudos
1 More Replies
Sk2
by • New Contributor
  • 1639 Views
  • 2 replies
  • 0 kudos

Getting error while mounting storage account to databricks notebook

Please help to reolve this issue i implemented all possible steps, credentials are correct and i am not using free trial account, i was watching tutorial to mount i used same code and everything is done as said in tutorial but iam getting this weird ...

Data Engineering
Reply
  • 1639 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
DeltaTrain
New Contributor II
  • 0 kudos

Hi, I am not so sure. But very likely you have tried with any cluster with Access mode as Shared.Can you try with a single node cluster, if the problem still exists, please share your problem once again here.please tag me in your question again if ne...

  • 0 kudos
1 More Replies
Madman
by • New Contributor II
  • 13518 Views
  • 5 replies
  • 6 kudos

Snowflake connection to Databricks error

When I am trying to read snowflake table from my databricks notebook, it is giving the error as:df1.read.format("snowflake") \.options(**options) \.option("query", "select * from abc") \.save()Getting below errorjava.sql.SQLException: No suitable dri...

Data Engineering
Reply
  • 13518 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
pdiegop
New Contributor II
  • 6 kudos

@anurag2192 did you managed to solve it?

  • 6 kudos
4 More Replies
Axatar
by • New Contributor III
  • 11367 Views
  • 3 replies
  • 3 kudos

Resolved! query takes too long to write into delta table.

hello, am running into in issue while trying to write the data into a delta table, the query is a join between 3 tables and it takes 5 minutes to fetch the data but 3hours to write the data into the table, the select has 700 records. here are the app...

Data Engineering
Reply
  • 11367 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Axatar
New Contributor III
  • 3 kudos

it turned out that the issue was not in the writing side, even when i was getting the results in under 5min, the issue was in the cross join in my query i resolved the issue by doing the same cross joins via dataframes got the results computed and wr...

  • 3 kudos
2 More Replies
Ajay-Pandey
by • Esteemed Contributor III
  • 4605 Views
  • 4 replies
  • 3 kudos

Resolved! Issue while re-starting Streaming DLT pipeline - PUBSUB

I am not able to restart our streaming DLT pipeline as it's giving "__tmp_path_dir" number format error, I am using PUB-SUB as a source for streaming.Any solutions will help - #pusub #databricks 

fff.jpg
Data Engineering
dlt
pubsub
streaming
Reply
  • 4605 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 3 kudos

 @Retired_mod @Amine 

  • 3 kudos
3 More Replies
elgeo
by • Valued Contributor II
  • 34657 Views
  • 9 replies
  • 5 kudos

Resolved! SQL Declare Variable equivalent in databricks

Hello. What would be the equivalent of the below in databricks?DECLARE @LastChangeDate as dateSET @LastChangeDate = GetDate()I already tried the below and worked. However I need to know how set a sql variable dynamicallySET da.dbname = test;SELECT "$...

Data Engineering
Reply
  • 34657 Views
  • 9 replies
  • 5 kudos
View Replies
Latest Reply
srinitechworld
New Contributor II
  • 5 kudos

hi try to to control the variables  

  • 5 kudos
8 More Replies
p4pratikjain
by • Contributor
  • 1649 Views
  • 1 replies
  • 0 kudos

Resolved! How to send Task Specific Parameters using run-now REST API.

I am using Following job trigger API -https://docs.databricks.com/api/workspace/jobs/runnowHere is what I want to achieve -I have complex workflow with multiple tasks and each task has python parameter named "--conf-file".While using runnow REST API ...

Data Engineering
Reply
  • 1649 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
victormarcel
New Contributor III
  • 0 kudos

@p4pratikjain wrote:I am using Following job trigger API - Popeyes Guest Experience Surveyhttps://docs.databricks.com/api/workspace/jobs/runnowHere is what I want to achieve - I have complex workflow with multiple tasks and each task has python param...

  • 0 kudos
Sivaji
by • New Contributor
  • 644 Views
  • 0 replies
  • 0 kudos

Data bricks Exam got suspended

Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam. I want to file a compla...

Data Engineering
Reply
  • 644 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels