cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Fz1
by • New Contributor III
  • 8569 Views
  • 6 replies
  • 3 kudos

Resolved! SQL Warehouse Serverless - Not able to access the external tables in the hive_metastore

I have DLT tables created under the hive_metastore with external data stored in ADL gen2.The ADL blob storage is mounted into /mnt/<storage-account>The tables are successfully created and accessible from my notebooks, as well the ADL storage.I have c...

Data Engineering
Reply
  • 8569 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
TjommeV-Vlaio
New Contributor III
  • 3 kudos

Can this be done using Terraform as well?

  • 3 kudos
5 More Replies
NimaiAhl
by • New Contributor II
  • 1200 Views
  • 1 replies
  • 0 kudos

External Tables - SQL

To create external tables we need to use the location keyword and use the link for the storage location, in reference to that does the user need to have permission for the storage location if not then will we use storage credentials to provide the ac...

Data Engineering
Reply
  • 1200 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Shikamaru
New Contributor II
  • 0 kudos

Hi Nimai, That's partially right. You can grant permissions directly on the storage credential, but Databricks recommends that you reference it in an external location and grant permissions to that instead. An external location combines a storage cre...

  • 0 kudos
Kopal
by • New Contributor II
  • 5235 Views
  • 3 replies
  • 3 kudos

Resolved! Data Engineering - CTAS - External Tables - Limitations of CTAS for external tables - can or cannot use options and location

Data Engineering - CTAS - External TablesCan someone help me understand why In chapter 3.3, we cannot not directly use CTAS with OPTIONS and LOCATION to specify delimiter and location of CSV?Or I misunderstood?Details:In Data Engineering with Databri...

Data Engineering
Reply
  • 5235 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

The 2nd statement CTAS will not be able to parse the csv in any manner because it's just the from statement that points to a file. It's more of a traditional SQL statement with select and from. It will create a Delta Table. This just happens to b...

  • 3 kudos
2 More Replies
ossinova
by • Contributor II
  • 1300 Views
  • 1 replies
  • 0 kudos

Schedule reload of system.information_schema for external tables in platform

Probably not feasible, but is there a way to update (via STORED PROCEDURE, FUNCTION or SQL query) the information schema of all external tables within Databricks. Last updated that I can see was when I converted the tables to Unity. From my understa...

Data Engineering
Reply
  • 1300 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Own
Contributor
  • 0 kudos

You can try optimize and cache with the internal tables such as schema tables to fetch updated information.

  • 0 kudos
ravikanthranjit
by • New Contributor III
  • 3444 Views
  • 6 replies
  • 14 kudos

Vacuum on external tables that we mount on ADLS

Want to know the best process of removal of files on ADLS after Optimize and Vacuum Dry run is completed

Data Engineering
Reply
  • 3444 Views
  • 6 replies
  • 14 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 14 kudos

Hi @Ravikanth Narayanabhatla​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...

  • 14 kudos
5 More Replies
RohitKulkarni
by • Contributor II
  • 4464 Views
  • 5 replies
  • 4 kudos

External table issue

Hello Team,I am using df.write command and the table is getting created. If you refer the below screenshot the table got created in Tables folder in dedicated sql pool. But i required in the External Tables folder. RegardsRK

image image
Data Engineering
Reply
  • 4464 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

if you actually write into Synapse, it is not an external table. the data resides on synapse.If you want to have an external table, write the data on your data lake in parquet/delta lake format and then create an external table on that location in s...

  • 4 kudos
4 More Replies
Labels