cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
ZacayDaushin
by • New Contributor
  • 855 Views
  • 2 replies
  • 0 kudos

How to access system.access.table_lineage

I try to make a select from system.access.table_lineage but i dont have to see the tablewhat permission to i have 

Data Engineering
Reply
  • 855 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
1 More Replies
AChang
by • New Contributor III
  • 3171 Views
  • 2 replies
  • 0 kudos

Resolved! Move a folder from Workspace to DBFS

So, I didn't quite set up my model training output directory correctly, and it saved all my model files to the workspace in the git repo I was working in. I am trying to move these files to DBFS, but when I try using dbutils.fs.mv, I get this error: ...

Data Engineering
Reply
  • 3171 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

That’s a great tip! Thank you for sharing your knowledge with the community. I’m sure many people will find it helpful. Keep up the good work!

  • 0 kudos
1 More Replies
Hardy
by • New Contributor III
  • 4265 Views
  • 7 replies
  • 3 kudos

upload files to dbfs:/volume using databricks cli

In our azure pipeline we are using databricks-cli command to upload jar files at dbfs:/FileStore location and that works perfectly fine. But when we try to use the same command to upload files at dbfs:/Volume/dev/default/files, it does not work and g...

Data Engineering
Reply
  • 4265 Views
  • 7 replies
  • 3 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 3 kudos
6 More Replies
taschi
by • New Contributor II
  • 9168 Views
  • 5 replies
  • 1 kudos

Resolved! How can I trigger the execution of a specific step within a Databricks Workflow job?

I'm investigating methods to test a Job starting from a particular step. For instance, if I've made modifications midway through a 50+ step Job, is there a way to test the Job without running the steps that precede the one with the modification?

Data Engineering
Reply
  • 9168 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 1 kudos
4 More Replies
Volker
by • New Contributor III
  • 1101 Views
  • 3 replies
  • 2 kudos

Preferred compression format for ingesting large amounts of JSON files with Autoloader

Hello Databricks Community,in an IOT context we plan to ingest a large amount of JSON files (~2 Million per Day). The JSON files are in json lines format und need to be compressed on the IOT devices. We can provide suggestions for the type of compres...

Data Engineering
Reply
  • 1101 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Volker
New Contributor III
  • 2 kudos

Hi, sorry I guess my response wasn't sent. The source are JSON files that are uploaded to an S3 bucket. The sink will be a Delta Table and we are using autoloader.The question was about the compression format of the incoming JSON files, e.g. if it wo...

  • 2 kudos
2 More Replies
manohar3
by • New Contributor III
  • 1700 Views
  • 4 replies
  • 0 kudos

Resolved! spark databricks jdbc driver integration return rows having column names as values

Hi all,i am using below to code to query table but query returns rows having column names as valuesspark.read .format("jdbc") .option("url", "jdbc:databricks://acme.cloud.databricks.com:443/myschema;transportMode=http;ssl=1;httpPath=<httppath>;Au...

Data Engineering
Reply
  • 1700 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

That’s a great tip! Thank you for sharing your knowledge with the community. I’m sure many people will find it helpful. Keep up the good work!

  • 0 kudos
3 More Replies
Manjusha
by • New Contributor II
  • 906 Views
  • 4 replies
  • 0 kudos

Failed to create notebook on community edition

Hi,I am unable to create new notebook on databricks community edition.  getting error 'failed to create notebook' when I click on  create-> notebookIs anyone else facing the same issue? if so, any tips on how to resolve it?

Data Engineering
Reply
  • 906 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Please look at this link related to the Community - Edition, which might solve your problem.   I appreciate your interest in sharing your Community-Edition query with us.   If you have any more questions or concerns, please don't hesitate to let us k...

  • 0 kudos
3 More Replies
learning_1989
by • New Contributor II
  • 1393 Views
  • 4 replies
  • 2 kudos

You have json file which is nested with multiple key value pair how you read it in databricks?

You have json file which is nested with multiple key value pair how you read it in databricks?

Data Engineering
Reply
  • 1393 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 2 kudos
3 More Replies
DumbBeaver
by • New Contributor II
  • 1151 Views
  • 3 replies
  • 1 kudos

Resolved! ERROR: Writing to Unity Catalog from Remote Spark using JDBC

This is my code here. df = spark.createDataFrame([[1,1,2]], schema=['id','first_name','last_name'])(df.write.format("jdbc")     .option("url",  <jdbc-url>)    .option("dbtable","hive_metastore.default.test")    .option("driver", "com.databricks.clien...

Data Engineering
Reply
  • 1151 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution. 

  • 1 kudos
2 More Replies
kazinahian
by • New Contributor III
  • 1333 Views
  • 3 replies
  • 3 kudos

Resolved! How can I Learn Databricks Data Pipeline in Azure environment?

Hello Esteemed Community,I have a fundamental question to ask, and I approach it with a sense of humility. Your guidance in my learning journey would be greatly appreciated. I am eager to learn how to build a hands-on data pipeline within the Databri...

Data Engineering
Reply
  • 1333 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

That’s a great tip! Thank you for sharing your knowledge with the community. I’m sure many people will find it helpful. Keep up the good work!

  • 3 kudos
2 More Replies
desertstorm
by • New Contributor II
  • 2026 Views
  • 7 replies
  • 0 kudos

Driver Crash on processing large dataframe

I have a dataframe with abt 2 million text rows (1gb). I partition it into about 700 parititons as thats the no of cores available on my cluster exceutors. I run the transformations extracting medical information and then write the results in parquet...

Data Engineering
Reply
  • 2026 Views
  • 7 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
6 More Replies
CloudPlatformer
by • New Contributor II
  • 1659 Views
  • 2 replies
  • 0 kudos

Npip Tunnel Setup Failure

Hi everyone,I'm currently running into an issue when trying to create any type of compute cluster in a workspace (premium, with VNet Injection and private DNS zone + private Endpoint). The operation always fails with: Compute terminated. Reason: Npip...

Data Engineering
Reply
  • 1659 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @CloudPlatformer, The NPIP tunnel setup failure during cluster creation in Databricks can be frustrating, but let’s troubleshoot it together.    Here are some steps to investigate the issue:   Network Configuration: Ensure that your network setup ...

  • 0 kudos
1 More Replies
jdodson2
by • New Contributor
  • 819 Views
  • 1 replies
  • 0 kudos

Need to reschedule my Databricks Data Engineer Professional Certification exam

I'm James Dodson and I was scheduled to take the Databricks Data Engineer Certification exam 1/30/2024 at 8:30am EST-Detroit.  My internet connection went down before I could take the exam.  Xfinity, my internet provider sent a SMS text stating there...

Data Engineering
Reply
  • 819 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).

  • 0 kudos
RyHubb
by • New Contributor III
  • 2539 Views
  • 6 replies
  • 0 kudos

Resolved! Databricks asset bundles job and pipeline

Hello, I'm looking to create a job which is linked to a delta live table.  Given the job code like this: my_job_name: name: thejobname schedule: quartz_cron_expression: 56 30 12 * * ? timezone_id: UTC pause_stat...

Data Engineering
Reply
  • 2539 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

I want to express my gratitude for your effort in selecting the most suitable solution. It's great to hear that your query has been successfully resolved. Thank you for your contribution. 

  • 0 kudos
5 More Replies
Kai
by • New Contributor II
  • 1502 Views
  • 2 replies
  • 0 kudos

Resolved! Differences Between "TEMPORARY STREAMING TABLE" and "TEMPORARY STREAMING LIVE VIEW" in DLT

Hello Databricks community,I'm seeking clarification on the distinctions between the following two syntaxes:CREATE OR REFRESH TEMPORARY STREAMING TABLECREATE TEMPORARY STREAMING LIVE VIEWAs of my understanding, both of these methods do not store data...

Data Engineering
Reply
  • 1502 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hey there! Thanks a bunch for being part of our awesome community!  We love having you around and appreciate all your questions. Take a moment to check out the responses – you'll find some great info. Your input is valuable, so pick the best solution...

  • 0 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all