cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Shanthala
by • New Contributor III
  • 1802 Views
  • 4 replies
  • 2 kudos

Resolved! Can employees of a partner can sign up directly to the Databricks Academy?

Can employees of a partner sign up directly to the Databricks Academy without singing up for Partner Portal - https://partners.databricks.com/ ?

Data Engineering
Reply
  • 1802 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Shanthala
New Contributor III
  • 2 kudos

thank you all!! got it. we started a cohort to study Databricks Engineering path last week by directly registering in Partner Academy!!

  • 2 kudos
3 More Replies
Anonymous
by • Not applicable
  • 1617 Views
  • 2 replies
  • 0 kudos

Safari errors in loading

Safari shows a white page, javascript console reports some errors: SyntaxError: Invalid regular expression: invalid group specifier nameUnhandled Promise Rejection: ChunkLoadError: Loading chunk 6956 failed.(missing: https://dd-databricks-live.cloud....

Data Engineering
Reply
  • 1617 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Could you please make sure the safari is updated? Also, please try to clear the caches , history and restart the browser. Also, what are you trying to open and this error is surfaced?

  • 0 kudos
1 More Replies
Joao_DE
by • New Contributor III
  • 1761 Views
  • 4 replies
  • 0 kudos

JDBC connection

Hi everyone! I have a question. For a project I need to establish a jdbc connection using spark.read. My question is when does the connection is deleted. That is because I will read multiple tables from that database, so if I could just create a conn...

Data Engineering
Reply
  • 1761 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Joao_DE
New Contributor III
  • 0 kudos

Hi Vidula!I haven´t figure out a solution yet, so any help would be appreciatedThank you!

  • 0 kudos
3 More Replies
Ayur
by • New Contributor II
  • 3467 Views
  • 3 replies
  • 4 kudos

Resolved! Unsupported_operation : Magic commands (e.g. %py, %sql and %run) are not supported with the exception of %pip within a Python notebook. Cells containing magic commands are ignored - DLT pipeline

Hi,I'm trying to use magic command(to change to python in a notebook with sql as a default language) in a dlt pipeline,.When starting the pipeline cells containing magic command are ignored., with the warning message below:"Magic commands (e.g. %py, ...

Data Engineering
Reply
  • 3467 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Yassine Dehbi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 4 kudos
2 More Replies
chanda02
by • New Contributor II
  • 1558 Views
  • 2 replies
  • 0 kudos

GRANT permission does not work on column level access control for table in unity catalog

I am trying to run below SQL command for granting column specific control to user but it throws error.I have given USE_CATALOG for catalog and USE_SCHEMA permission for schema for the user and have set current catalog and schema.I am using Databricks...

image
Data Engineering
Reply
  • 1558 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, you can find the best practises here: https://docs.databricks.com/data-governance/unity-catalog/best-practices.htmlAlso, for cluster access modes for unity catalog : https://docs.databricks.com/data-governance/unity-catalog/index.html#cluster-sec...

  • 0 kudos
1 More Replies
Leszek
by • Contributor
  • 809 Views
  • 1 replies
  • 3 kudos

How to handle schema changes in streaming Delta Tables?

I'm using Structure Streaming when moving data from one Delta Table to another.How to handle schema changes in those tables (e.g. adding new column)?

Data Engineering
Reply
  • 809 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Murthy1
Contributor II
  • 3 kudos

Hello,I think the only way of handling is to mention the schema within the job through a schema file. The other way is to restart the job to infer the new schema automatically.

  • 3 kudos
KevSpally
by • New Contributor
  • 1087 Views
  • 1 replies
  • 0 kudos

When accessing a view in Unity Catalog; access to underlying tables of the view is also needed.

My goal is that i want to provide users access to a view but not the underlying tables. I only want them to see specific columns and rows of the table. When i just give the select permissions on the view the user gets an error that they also need acc...

Data Engineering
Reply
  • 1087 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
jonathan_ruiz
New Contributor II
  • 0 kudos

I have exactly the same question, did anyone get the right answer?

  • 0 kudos
chanansh
by • Contributor
  • 1186 Views
  • 1 replies
  • 0 kudos

Resolved! autoloader documentation does not work

I am trying to following the documentation here:https://learn.microsoft.com/en-us/azure/databricks/getting-started/etl-quick-startMy code looks like:(spark.readStream .format("cloudFiles") .option("header", "true") #.option("cloudFiles.partitio...

Data Engineering
Reply
  • 1186 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Murthy1
Contributor II
  • 0 kudos

Hi,It seems like you are writing to a path which is not empty and has some non - delta format files. Also, can you confirm if the path mentioned in the error message "`s3://nbu-ml/projects/rca/msft/dsm09collectx/delta` " is the path you are writing t...

  • 0 kudos
MichaelN1
by • New Contributor II
  • 5035 Views
  • 6 replies
  • 8 kudos

How to change the width of the lines for Python Format in a notebook?

Hidoes anyone know how to change the default 80 symbols for the line length when executing "Format Python Code" in Databricks notebook?These days ultra wide monitors are new standards and having 80 line width with breaks to a new line is not usable

Data Engineering
Reply
  • 5035 Views
  • 6 replies
  • 8 kudos
View Replies
Latest Reply
Rajeev_Basu
Contributor III
  • 8 kudos

It may not be possible presently. but would be good to be recommended to Databricks as a feedback.

  • 8 kudos
5 More Replies
Rishabh-Pandey
by • Honored Contributor III
  • 7534 Views
  • 7 replies
  • 4 kudos

Resolved! connect databricks to teradata

hey i want to know can we connect databricks to the teradata database and if yes what will be the procedure ??? help would be appreciated

Data Engineering
Reply
  • 7534 Views
  • 7 replies
  • 4 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 4 kudos

use the JDBC driver from here https://docs.databricks.com/integrations/jdbc-odbc-bi.html

  • 4 kudos
6 More Replies
venkat-bodempud
by • New Contributor III
  • 3770 Views
  • 4 replies
  • 7 kudos

Power BI - Databricks Integration using Service Principal

Hello Community,We are able to connect to databricks(using Personal access token) from Power BI Desktop and we able to set up scheduling databricks notebook using DataFactory for every 10 minutes(as per our requirement). We want to avoid using the pe...

Data Engineering
Reply
  • 3770 Views
  • 4 replies
  • 7 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 7 kudos

You can use the token generated for the service principal and use it. As a security best practice, when authenticating with automated tools, systems, scripts, and apps, Databricks recommends you use access tokens belonging to service principals inste...

  • 7 kudos
3 More Replies
chanansh
by • Contributor
  • 1205 Views
  • 2 replies
  • 0 kudos

how to compute difference over time of a spark structure streaming?

I have a table with a timestamp column (t) and a list of columns for which I would like to compute the difference over time (v), by some key(k): v_diff(t) = v(t)-v(t-1) for each k independently.Normally I would write:lag_window = Window.partitionBy(C...

Data Engineering
Reply
  • 1205 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
chanansh
Contributor
  • 0 kudos

I found this but could not make it work https://www.databricks.com/blog/2022/10/18/python-arbitrary-stateful-processing-structured-streaming.html

  • 0 kudos
1 More Replies
Skesaram
by • New Contributor II
  • 869 Views
  • 1 replies
  • 0 kudos

Need help to connect to local DB from Data bricks

jdbcHostname="478"jdbcPort=1433jdbcDatabase="Onprem_AzureDB"jdbcUsername="upendra"jdbcPassword="upendrakumar"jdbcDriver="com.microsoft.sqlserver.jdbc.SQLServerDriver"jdbcUrl=f"jdbc:sqlserver://{jdbcHostname}:{jdbcPort};databaseName={jdbcDatabase};use...

Data Engineering
Reply
  • 869 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Could you please verify the network connectivity from Databricks to the SQL server? Please make sure SQL:port is allowed in your firewall rules or security groups.

  • 0 kudos
lasmali
by • New Contributor II
  • 1589 Views
  • 3 replies
  • 0 kudos

Instance Profile creation via the Databricks REST API returns "INVALID_PARAMETER_VALUE"

# Problem StatementWe have a need to create Instance Profiles via the Databricks REST API, but the endpoint returns an "INVALID_PARAMETER_VALUE" error with "Syntactically invalid AWS instance profile ARN" message even when provided an appropriate ARN...

Data Engineering
Reply
  • 1589 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @lasse Lidegaard​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 0 kudos
2 More Replies
mebinjoy
by • New Contributor II
  • 2681 Views
  • 6 replies
  • 8 kudos

Resolved! Certificate not received.

I had completed the Data Engineering Associate V3 certification today morning and I'm yet to receive my certification. I had received a mail stating that I had passed and the certification would be mailed.

Data Engineering
Reply
  • 2681 Views
  • 6 replies
  • 8 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 8 kudos

Hi @Mebin Joy​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. Regards

  • 8 kudos
5 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors
View all