cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Ajay-Pandey
by Esteemed Contributor III
  • 26296 Views
  • 6 replies
  • 7 kudos

Resolved! What does "Determining location of DBIO file fragments..." mean, and how do I speed it up?

Determining location of DBIO file fragments. This operation can take some time.What does this mean, and how do I prevent it from having to perform this apparently-expensive operation every time? This happens even when all the underlying tables are De...

  • 26296 Views
  • 6 replies
  • 7 kudos
Latest Reply
Christianben9
New Contributor II
  • 7 kudos

Determining location of DBIO file fragments" is a message that may be displayed during the boot process of a computer running the NetApp Data ONTAP operating system. This message indicates that the system is currently in the process of identifying an...

  • 7 kudos
5 More Replies
cgrant
by Databricks Employee
  • 3722 Views
  • 4 replies
  • 6 kudos

How do I know how much of a query/job used Photon?

I'm trying to use the native execution engine, Photon. How can I tell if a query is using Photon or is falling back to the non-native Spark engine?

  • 3722 Views
  • 4 replies
  • 6 kudos
Latest Reply
venkat09
New Contributor III
  • 6 kudos

Typo error in my second point of the previous post. Click the execution plan of your task[this is available under SQL/Dataframe tab in Spark UI]. It explains what operations run in the photon engine and what didn't execute by photon.

  • 6 kudos
3 More Replies
patdev
by New Contributor III
  • 6575 Views
  • 9 replies
  • 2 kudos

text datatype not supported and data having huge data in text filed how to bring it over

Hello all,I have medical field data file and one of the field is the text field with huge data not the big problem is databrick does not support text data type so how can i bring the data over. i tried conversion, cast in various way but so far not ...

  • 6575 Views
  • 9 replies
  • 2 kudos
Latest Reply
patdev
New Contributor III
  • 2 kudos

Setting escapeQuotes to false has helped to bring huge text data in colomn.thanks

  • 2 kudos
8 More Replies
Sujitha
by Databricks Employee
  • 862 Views
  • 1 replies
  • 2 kudos

Documentation Update January 13 - 19 Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, a...

Documentation Update January 13 - 19Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, ...

  • 862 Views
  • 1 replies
  • 2 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

thanks for the details

  • 2 kudos
elgeo
by Valued Contributor II
  • 8387 Views
  • 2 replies
  • 3 kudos

Resolved! Trigger on a table

Hello! Is there an equivalent of Create trigger on a table in Databricks sql?CREATE TRIGGER [schema_name.]trigger_nameON table_nameAFTER {[INSERT],[UPDATE],[DELETE]}[NOT FOR REPLICATION]AS{sql_statements}Thank you in advance!

  • 8387 Views
  • 2 replies
  • 3 kudos
Latest Reply
AdrianLobacz
Contributor
  • 3 kudos

You can try Auto Loader: Auto Loader supports two modes for detecting new files: directory listing and file notification.Directory listing: Auto Loader identifies new files by listing the input directory. Directory listing mode allows you to quickly ...

  • 3 kudos
1 More Replies
Raghu101
by New Contributor III
  • 5670 Views
  • 6 replies
  • 3 kudos

How to Call Oracle Stored Procedures from Databricks?

How to Call Oracle Stored Procedures from Databricks?

  • 5670 Views
  • 6 replies
  • 3 kudos
Latest Reply
sher
Valued Contributor II
  • 3 kudos

https://datathirst.net/blog/2018/10/12/executing-sql-server-stored-procedures-on-databricks-pyspark/try this link. this may help you

  • 3 kudos
5 More Replies
gbradley145
by New Contributor III
  • 4672 Views
  • 3 replies
  • 4 kudos

Why does Databricks SQL drop ending 0 in decimal data type

All,I have a column, RateAdj that is defined as DECIMAL(15,5) and I can see that the value is 4.00000, but when this gets inserted into my table it shows as just 4.%sql   SELECT LTRIM(RTRIM(IFNULL(FORMAT_NUMBER(RateADJ, '0.00000'), '0.00000')))This i...

  • 4672 Views
  • 3 replies
  • 4 kudos
Latest Reply
silvathomas
New Contributor II
  • 4 kudos

The value goes to 10,000 values and having the things done to run a fast execution, and I am also Sociology Dissertation Help with the reduction of pages.

  • 4 kudos
2 More Replies
vanessafvg
by New Contributor III
  • 3596 Views
  • 3 replies
  • 1 kudos

linking filters from different Databricks SQL queries in a Dashboard

I am having to use Databricks SQL dashboard for some analysis, it seems very clunky. If i have multiple queries, is it possible to apply the same filters to all the queries in the dashboard or do i have to duplicate the filters for each query in the ...

  • 3596 Views
  • 3 replies
  • 1 kudos
Latest Reply
FelixH
New Contributor II
  • 1 kudos

Same issue here. According the docs, using query filters with the same name and values should result in a single dashboard filter. However, filters are duplicated. I also tried using this setting but no success

  • 1 kudos
2 More Replies
vr
by Contributor
  • 5803 Views
  • 3 replies
  • 3 kudos

Resolved! Is timestamp difference always INTERVAL DAY TO SECOND?

My observations show that timestamp difference has type of INTERVAL DAY TO SECONDS:select typeof(getdate() - current_date()) ----------------------------------------- interval day to secondBut is it guaranteed? Can it be DAY TO MINUTE or, say, YEAR T...

  • 5803 Views
  • 3 replies
  • 3 kudos
Latest Reply
sher
Valued Contributor II
  • 3 kudos

you can check here for given example: https://docs.databricks.com/sql/language-manual/functions/minussign.htmlthis might help to you.

  • 3 kudos
2 More Replies
prasannar
by New Contributor II
  • 3581 Views
  • 3 replies
  • 3 kudos
  • 3581 Views
  • 3 replies
  • 3 kudos
Latest Reply
sher
Valued Contributor II
  • 3 kudos

Df.write.format('jdbc').options( url='jdbc:oracle:thin:@192.168.11.100:1521:ORCL', driver='oracle.jdbc.driver.OracleDriver', dbtable='testschema.test', user='testschema', password='password').mode('overwrite').save()try ...

  • 3 kudos
2 More Replies
dulu
by New Contributor III
  • 12258 Views
  • 5 replies
  • 6 kudos

split character string in cell with sql

I have the following input: I am looking for a way to split the characters in the item_order_detail column into 2 columns itemID and itemName. As below output table uses SQL function in databricks with spark_sql version 3.2.1.Can someone suggest a so...

hinh22 hinh223
  • 12258 Views
  • 5 replies
  • 6 kudos
Latest Reply
sher
Valued Contributor II
  • 6 kudos

you need to use explode functionhttps://stackoverflow.com/questions/61070630/spark-explode-column-with-json-array-to-rows

  • 6 kudos
4 More Replies
VVill_T
by Contributor
  • 3955 Views
  • 4 replies
  • 7 kudos

How to write a Delta Live Table(dlt) pipeline output to Databricks SQL directly

Hi,I am trying to see if it is possible to setup a direct connection from dlt pipeline to a table in Databricks SQL by configuring the Target Schema: with poc being a location of schema like "dbfs:/***/***/***/poc.db The error message was just a...

image image
  • 3955 Views
  • 4 replies
  • 7 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 7 kudos

When ever you store a Delta Table to Hive Metastore. This table will be available in Databricks SQL Workspace ( Data Explorer ) under hive_metastore catalog.

  • 7 kudos
3 More Replies
KVNARK
by Honored Contributor II
  • 3531 Views
  • 4 replies
  • 13 kudos

Resolved! To practice Databricks SQL

Is there any sand box kind of thing where we can do some hands-on on Databricks SQL/run the Note books attaching to the Clusters apart from the free trial provided by Databricks.

  • 3531 Views
  • 4 replies
  • 13 kudos
Latest Reply
Harun
Honored Contributor
  • 13 kudos

Databricks SQL workspace will be available only for Databricks Premium service. If you have Azure Pass subscription, then you can able to get it for practicing it.

  • 13 kudos
3 More Replies
avidex180899
by New Contributor II
  • 11539 Views
  • 3 replies
  • 3 kudos

Resolved! UUID/GUID Datatype in Databricks SQL

Hi all,I am trying to create a table with a GUID column.I have tried using GUID, UUID; but both of them are not working.Can someone help me with the syntax for adding a GUID column?Thanks!

  • 11539 Views
  • 3 replies
  • 3 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

Hey @Avinash Narasimhan​ , What is the exact problem you are getting can you please share it is working fine for meThanksAviral Bhardwaj

  • 3 kudos
2 More Replies
Wout
by Contributor
  • 7432 Views
  • 6 replies
  • 7 kudos

Resolved! Wrong X-Axis Order when Visualization is Put on Dashboard

I have a visualization in which the X-axis values are displayed correctly in the Query Editor, in the order produced by the SQL query. However, when I add the visualization to a dashboard, the values are suddenly not sorted anymore.How is this possib...

correct wrong
  • 7432 Views
  • 6 replies
  • 7 kudos
Latest Reply
Wout
Contributor
  • 7 kudos

We have further analyzed the visualization problem and found two solutions.The original visualization consists of 1 series and has aggregation enabled in the UI (but is unused, since the query itself aggregates already).We found that the following tw...

  • 7 kudos
5 More Replies
Labels