cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
jwilliam
by • Contributor
  • 2275 Views
  • 2 replies
  • 1 kudos

Resolved! [BUG] Databricks install WHL as JAR in Python Wheel Task?

I'm using Python Wheel Task in Databricks job with WHEEL dependencies. However, the cluster installed the dependencies as JAR instead of WHEEL. Is this an expected behavior or a bug?

Data Engineering
Reply
  • 2275 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
AndréSalvati
New Contributor III
  • 1 kudos

There you can see a complete template project with a python wheel task and Databricks Asset Bundles. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template

  • 1 kudos
1 More Replies
Twilight
by • New Contributor III
  • 5181 Views
  • 5 replies
  • 3 kudos

Resolved! Bug - Databricks requires extra escapes in repl string in regexp_replace (compared to Spark)

In Spark (but not Databricks), these work:regexp_replace('1234567890abc', '^(?<one>\\w)(?<two>\\w)(?<three>\\w)', '$3$2$1') regexp_replace('1234567890abc', '^(?<one>\\w)(?<two>\\w)(?<three>\\w)', '${three}${two}${one}')In Databricks, you have to use ...

Data Engineering
Reply
  • 5181 Views
  • 5 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@Stephen Wilcoxon​ : No, it is not a bug. Databricks uses a different flavor of regular expression syntax than Apache Spark. In particular, Databricks uses Java's regular expression syntax, whereas Apache Spark uses Scala's regular expression syntax....

  • 3 kudos
4 More Replies
Rik
by • New Contributor III
  • 2220 Views
  • 2 replies
  • 0 kudos

Incorrect error when adding an IP access list

I have disabled the IP Access List on my workspace and am trying to add an IP list through the IP Access List API. However, when adding a list, I get the INVALID_STATE response.The docs mention this is because:"If the new list would block the calling...

Data Engineering
Reply
  • 2220 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Rik
New Contributor III
  • 0 kudos

"One possible workaround could be to (1) temporarily enable the IP Access List feature, (2) add the necessary IP addresses to the list, and then (3) disable the feature again. This way, you can add the IP addresses you need without blocking the curre...

  • 0 kudos
1 More Replies
haggholm
by • New Contributor
  • 2471 Views
  • 2 replies
  • 1 kudos

Resolved! Query with ORDER BY fails with HiveThriftServerError "requirement failed: Subquery … has not finished"

Using ODBC or JDBC to read from a table fails when I attempt to use an ORDER BY clause. In one sample case, I have a fairly small table (just 1946 rows).select * from some_table order by some_fieldResult:java.lang.IllegalArgumentException: requiremen...

Data Engineering
Reply
  • 2471 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @petter@hightouch.com Petter​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it doe...

  • 1 kudos
1 More Replies
desert_safari
by • New Contributor II
  • 2100 Views
  • 2 replies
  • 0 kudos

Bugs with text query parameters?

Hi all,I have query with a handful of text query parameters, I need to use to insert data into a table from a dashboard e.g.INSERT INTO user_data (first_name, middle_name, last_name, city, country, zip_code) VALUES ('{{first_name}}', '{{middle_name}}...

Data Engineering
Reply
  • 2100 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
desert_safari
New Contributor II
  • 0 kudos

Sorry my client wants all 3 fields. The middle name has to be there. There are a few cases where people have just one name.

  • 0 kudos
1 More Replies
Christine
by • Contributor II
  • 10134 Views
  • 1 replies
  • 2 kudos

ADD COLUMN IF NOT EXISTS does not recognize "IF NOT EXIST". How do I add a column to an existing delta table with SQL if the column does not already exist?

How do I add a column to an existing delta table with SQL if the column does not already exist?I am using the following code: <%sqlALTER TABLE table_name ADD COLUMN IF NOT EXISTS column_name type; >but it prints the error: <[PARSE_SYNTAX_ERROR] Synta...

Data Engineering
Reply
  • 10134 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
UmaMahesh1
Honored Contributor III
  • 2 kudos

Hi @Christine Pedersen​ I guess IF NOT EXISTS or IF EXISTS can be used in conjunction with DROP or PARTITIONS according to the documentation. If you want to do this the same checking way, you can do using a try catch block in pyspark or as per your l...

  • 2 kudos
rv1
by • New Contributor
  • 4158 Views
  • 0 replies
  • 0 kudos

NULL vs NaN in SQL Mode

In SQL Mode | SQL Editor there seems to be no distinction between NULL and NaN. In some cases it is very misleading as it makes the user to search mistake in the wrong place.DE/DS mode works as expected: UPDATE: a bit later I found this article: http...

image image
Data Engineering
Reply
  • 4158 Views
  • 0 replies
  • 0 kudos
KrishZ
by • Contributor
  • 1114 Views
  • 2 replies
  • 0 kudos

Where to report a bug with Databricks ?

I have in issue in Pyspark.Pandas to report. Is there a github or some forum where I can register my issue?Here's the issue

Data Engineering
Reply
  • 1114 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Does @Debayan Mukherjee​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
KrishZ
by • Contributor
  • 2487 Views
  • 2 replies
  • 1 kudos

Where to report a bug with Databricks ?

I have in issue in Pyspark.Pandas to report.Is there a github or some forum where I can register my issue?Here's the issue

Data Engineering
Reply
  • 2487 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, @Krishna Zanwar​ Could you please raise a support case to report the bug. Please refer https://docs.databricks.com/resources/support.html to engage with Databricks Support.

  • 1 kudos
1 More Replies
chris_kimmel
by • New Contributor II
  • 899 Views
  • 0 replies
  • 2 kudos

Bug report: Switching branches duplicates cells

I'm using Databricks' support for GitHub repos. When I switch from one branch to another while a notebook is open, it messes up my notebook. Specifically, every notebook cell appears twice after switching branches.

Data Engineering
Reply
  • 899 Views
  • 0 replies
  • 2 kudos
Emiel_Smeenk
by • New Contributor III
  • 13320 Views
  • 5 replies
  • 8 kudos

Resolved! Databricks Runtime 10.4 LTS - AnalysisException: No such struct field id in 0, 1 after upgrading

Hello,We are working to migrate to databricks runtime 10.4 LTS from 9.1 LTS but we're running into weird behavioral issues. Our existing code works up until runtime 10.3 and in 10.4 it stopped working.Problem:We have a nested json file that we are fl...

image image image
Data Engineering
Reply
  • 13320 Views
  • 5 replies
  • 8 kudos
View Replies
Latest Reply
Emiel_Smeenk
New Contributor III
  • 8 kudos

It seems like the issue was miraculously resolved. I did not make any code changes but everything is now running as expected. Maybe the latest runtime 10.4 fix released on April 19th also resolved this issue unintentionally.

  • 8 kudos
4 More Replies
MohitAnchlia
by • New Contributor II
  • 1074 Views
  • 0 replies
  • 1 kudos

Change AWS storage setting and account

I am seeing a super weird behaviour in databricks. We initially configured the following: 1. Account X in Account Console -> AWS Account arn:aws:iam::X:role/databricks-s3 2. We setup databricks-s3 as S3 bucket in Account Console -> AWS Storage 3. W...

Data Engineering
Reply
  • 1074 Views
  • 0 replies
  • 1 kudos
Labels