cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

harikrishnang33
by New Contributor II
  • 1814 Views
  • 2 replies
  • 2 kudos

Perform database operations like INSERT, and MERGE using the backend service (golang)

I would like to have an example or reference documentation on how we can create a live (capable of real-time data persistence) pipeline from an in-house golang (backend) service to the databricks tables.I would give a bit more detail on what the gola...

  • 1814 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Harikrishnan G​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 2 kudos
1 More Replies
Lakshmi_J
by New Contributor II
  • 4491 Views
  • 2 replies
  • 3 kudos

Unable to read data from delta table using a python script after the table properties change.

Renamed the Column to include () in a delta table and set the table properties to the below ​ALTER TABLE test_table SET TBLPROPERTIES (  'delta.minReaderVersion' = '2',  'delta.minWriterVersion' = '5',  'delta.columnMapping.mode' = 'name' ) However w...

Error
  • 4491 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Lakshmi Jayaraman​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 3 kudos
1 More Replies
FirozAhmad
by New Contributor II
  • 1203 Views
  • 3 replies
  • 1 kudos

unable to login Databricks partner account same id and password as Acadmey Account.

Unable to login Databric partner academy account. same as Acadmey give me a link to verify my mail to databric partner Account. I have try soo many time but unable to verify and login to databric partner Account .

  • 1203 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Firoz Ahmad Ansari​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly.

  • 1 kudos
2 More Replies
Jonas89
by New Contributor
  • 3770 Views
  • 2 replies
  • 0 kudos

Databricks Devops Release Pipeline Abort

We've built a release pipeline to our Databricks Workspaces, using the VNET Template. It's working end-to-end but intermittent aborts are occurring when workspace is recreated.For example, 4th of April (Monday) We recreated the workspaces and no abor...

1_PipelineError 2_PipelineError 3_PipelineError 4_PipelineError
  • 3770 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jonas Oliveira de Souza​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that be...

  • 0 kudos
1 More Replies
girish989
by New Contributor II
  • 821 Views
  • 2 replies
  • 0 kudos

Where is the dbc file for this couse : Deep Dive Into Lakehouse with Delta Lake

Where is the dbc file for this couse :Deep Dive Into Lakehouse with Delta Lake

  • 821 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @G S​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. 

  • 0 kudos
1 More Replies
Soma
by Valued Contributor
  • 831 Views
  • 2 replies
  • 1 kudos

Resolved! Where does custom state store the data

There are couple of custom state functions like mapgroupswithstate,ApplyinpandaswithStateWhich has a internal state maintained is it maintained in same statestore(rocksdb) as aggregation state store function ​

  • 831 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @somanath Sankaran​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 1 kudos
1 More Replies
ArunSharma
by New Contributor II
  • 3291 Views
  • 2 replies
  • 1 kudos

Database Objects Naming Convention for Bronze, Silver and Gold Layers

Please help me for database Objects Naming Convention and coding standard for Bronze, Silver and Gold Layers 

  • 3291 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Arun Sharma​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 1 kudos
1 More Replies
Leszek
by Contributor
  • 5022 Views
  • 5 replies
  • 5 kudos

Resolved! Unity Catalog - Azure account console - how to access?

I'm trying to access account console in Azure but I only can see the list of workspaces and access them. I didn't find documentation about account console for Azure. Do you know how to access account console?

  • 5022 Views
  • 5 replies
  • 5 kudos
Latest Reply
vimalii
New Contributor II
  • 5 kudos

Hello @Leszek​ . Please tell me is it works for you ?Did you find the root cause ?I still don't understand why I should grant to myself some extra permissions if I already global administrator, owner of subscription, owner of databricks workspace but...

  • 5 kudos
4 More Replies
Gajji
by New Contributor
  • 1577 Views
  • 2 replies
  • 0 kudos

How to link 2 Dashboards

Hi, I have 2 dashboards. 1st Dashboard has a table with two columns, 'Country' and 'NumberOfMajorCities'. When I click on any row of this table/click on 'NumberOfMajorCities' field of a row, I would like to jump to 2nd Dashboard, where I display all ...

  • 1577 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Ghazanfar Sl​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 0 kudos
1 More Replies
NM447101
by New Contributor II
  • 1909 Views
  • 3 replies
  • 1 kudos

Error when creating a delta live table pipeline

INVALID_PARAMETER_VALUE: Validation failed for node_type_id, the value must be Standard_DS3_v2 (is "Standard_F8s") 

image
  • 1909 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Nitya Mehta​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...

  • 1 kudos
2 More Replies
Teja07
by New Contributor II
  • 2379 Views
  • 4 replies
  • 0 kudos

Resolved! Datatype mismatch while reading data from sql server to databricks

Data from Azure sql server was read into databricks through JDBC connection (spark version 2.x) and stored into Gen1. Now the client wants to migrate the data from Gen1 to Gen2. When we ran the same jobs that read data from Azure Sql Server to Databr...

  • 2379 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Mani Teja G​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...

  • 0 kudos
3 More Replies
VeeruK
by New Contributor III
  • 1539 Views
  • 6 replies
  • 0 kudos

Databricks Lakehouse Fundamentals BadgeI have successfully passed the test after completion of the course "Databricks Lakehouse Fundamentals&quot...

Databricks Lakehouse Fundamentals BadgeI have successfully passed the test after completion of the course "Databricks Lakehouse Fundamentals". But I have'nt recieved any badge. I have been provided with a certificate only. Please provide me with th...

  • 1539 Views
  • 6 replies
  • 0 kudos
Latest Reply
VeeruK
New Contributor III
  • 0 kudos

Any update on the on the badge?

  • 0 kudos
5 More Replies
david_bernstein
by New Contributor III
  • 2300 Views
  • 6 replies
  • 0 kudos

DLT autoloader credentials not available error in Azure

I'm just getting started with Databricks and DLTs. I've followed all the docs and tutorials I can find on this and believe I have set up everything in Azure correctly: service principal, paths, and spark configs. When I run a simple DLT autoloader pi...

  • 2300 Views
  • 6 replies
  • 0 kudos
Latest Reply
david_bernstein
New Contributor III
  • 0 kudos

Thank you, I will look into this.

  • 0 kudos
5 More Replies
Fed
by New Contributor III
  • 2980 Views
  • 4 replies
  • 1 kudos

Resolved! Ray dashboard no longer available

Has anyone else experienced the lack of access to the Ray dashboard since this week? Last week worked fine.rom ray.util.spark import setup_ray_cluster   setup_ray_cluster(...)This used to output an HTML block with a link to the dashboard.I can manual...

  • 2980 Views
  • 4 replies
  • 1 kudos
Latest Reply
Fed
New Contributor III
  • 1 kudos

The reason for the missing dashboard for me was due ​to not having installed some required dependencies. Shout out to the Ray community for their help.​I've submitted a PR (now merged) to add a warning message​ when such dependencies are missing.​​

  • 1 kudos
3 More Replies
kj1
by New Contributor III
  • 3303 Views
  • 8 replies
  • 0 kudos

When running DBT pipeline with column docs persisted we get error at least one column must be specified

Problem:When running dbt with persist column docs enabled we get the following error: org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the tableBackground:There is an issue on the dbt-spark github that was c...

  • 3303 Views
  • 8 replies
  • 0 kudos
Latest Reply
Dooley
Valued Contributor
  • 0 kudos

Also confirming that you do not have any of these limitations:From DBT's website: Some databases limit where and how descriptions can be added to database objects. Those database adapters might not support persist_docs, or might offer only partial su...

  • 0 kudos
7 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels