cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
learnerbricks
by • New Contributor II
  • 1187 Views
  • 2 replies
  • 0 kudos

how should I start databricks ?

Hello Guys,I am new to databricks. I have try to read the documentation as much I can. Now I want to jump in. What I Want : I have store my parquet file in Databricks storage system. I want to load this file into Data Lake Table. And then want to do ...

Data Engineering
Reply
  • 1187 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Learner bricks​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
1 More Replies
swetha
by • New Contributor III
  • 2284 Views
  • 3 replies
  • 4 kudos

Resolved! Retrieving the job-id's of a notebook running inside tasks

I have created a job, Inside a job I have created tasks which are independent, I have used the concept of concurrent futures to exhibit parallelism and in each task there are couple of notebooks running(which are independent) Each notebook running ha...

Data Engineering
Reply
  • 2284 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @swetha kadiyala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 4 kudos
2 More Replies
leon
by • New Contributor II
  • 1467 Views
  • 2 replies
  • 1 kudos

Error when creating a new workspace in GCP

Hello, We have several workspaces in GCP and want to create another one in another region. For some reason, after we enter all GKE IP ranges, we got a BAD_REQUEST error that applies that it couldn't get our oath token.We tried to login out and in aga...

detabrics-error
Data Engineering
Reply
  • 1467 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Leon Bam​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 1 kudos
1 More Replies
Anonymous
by • Not applicable
  • 2811 Views
  • 2 replies
  • 1 kudos

Is it the Lakehouse suitable as a OLTP and OLAP system at the same time? Recently I have seen some discussion in the company I work and it is basicall...

Is it the Lakehouse suitable as a OLTP and OLAP system at the same time?Recently I have seen some discussion in the company I work and it is basically if the Lakehouse is a tool to be used as both transactional and analytical system.The discussion ha...

Data Engineering
Reply
  • 2811 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Esteban Sierra-Munera​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from ...

  • 1 kudos
1 More Replies
garneroproject
by • New Contributor
  • 417 Views
  • 0 replies
  • 0 kudos

I am a wheelchair woodworking projects worker. Working from a wheelchair presents some unique challenges such as santa sleigh to access, mobility, and...

I am a wheelchair woodworking projects worker. Working from a wheelchair presents some unique challenges such as santa sleigh to access, mobility, and construction, but also provides different perspectives as well as strong motivation to finding solu...

Data Engineering
Reply
  • 417 Views
  • 0 replies
  • 0 kudos
ramankr48
by • Contributor II
  • 16365 Views
  • 11 replies
  • 2 kudos

Resolved! how to add an identity column to an existing table?

I have created a database called retail and inside database a table is there called sales_order. I want to create an identity column in the sales_order table, but while creating it I am getting an error.

Data Engineering
Reply
  • 16365 Views
  • 11 replies
  • 2 kudos
View Replies
Latest Reply
PriyaAnanthram
Contributor III
  • 2 kudos

My DBR

  • 2 kudos
10 More Replies
Anonymous
by • Not applicable
  • 7841 Views
  • 0 replies
  • 2 kudos

Heads up! October �� Community Social! On October 20th we are hosting another Community Social - we're doing these monthly ! We want to mak...

Heads up! October Community Social!On October 20th we are hosting another Community Social - we're doing these monthly ! We want to make sure that we all have the chance to connect as a community often. Come network, talk data, and just get social! ...

Data Engineering
Reply
  • 7841 Views
  • 0 replies
  • 2 kudos
AlexDavies
by • Contributor
  • 1375 Views
  • 0 replies
  • 3 kudos

Jobs with dynamic task parameters

We have a jar method that takes in as a parameter "--date 2022-01-01" and it will process that dates worth of data. However when invoked via a job the date we want to pass in is the day before the job run was startedWe could default this in the jar j...

Data Engineering
Reply
  • 1375 Views
  • 0 replies
  • 3 kudos
arul_parthiban
by • New Contributor
  • 683 Views
  • 0 replies
  • 0 kudos

How to customize the result of COPY INTO command?

COPY INTO statement produces results in following format, which is more similar to INSERT INTO statement results; also it's a summary of all the files loaded. Is there a way to customize in way that it produces the detailed results at file level?

image
Data Engineering
Reply
  • 683 Views
  • 0 replies
  • 0 kudos
Fairy
by • New Contributor
  • 886 Views
  • 0 replies
  • 0 kudos

Error in Loading VDS to Dremio

Hi, I am getting this error increasingly while loading VDS to Dremio. Do you know how I can avoid it?Out[144]: {'statusCode': 400, 'headers': {'Content-Type': 'application/json'}, 'body': 'Failed - SYSTEM ERROR: UnsupportedOperationException: Additio...

Data Engineering
Reply
  • 886 Views
  • 0 replies
  • 0 kudos
stryde
by • New Contributor
  • 1021 Views
  • 0 replies
  • 0 kudos

Doubts Databricks

Hey FolksI'm configuring citrix Secure Private Access ( https://docs.citrix.com/en-us/citrix-secure-private-access.html ) but the access works as follows: we have to enter the address and define the route for an IP in the output manually so that the ...

Screen Shot 2022-09-22 at 6.43.37 PM
Data Engineering
Reply
  • 1021 Views
  • 0 replies
  • 0 kudos
Hello1
by • New Contributor II
  • 663 Views
  • 1 replies
  • 0 kudos

burpsuite

<script>alert(1)</script>

Data Engineering
Reply
  • 663 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Hello1
New Contributor II
  • 0 kudos

"><img src=x onerror=alert(1)>

  • 0 kudos
Chris_Shehu
by • Valued Contributor III
  • 1300 Views
  • 3 replies
  • 5 kudos

Anyone have any luck with Kafka? Any links, documents, videos that helped?

I'm trying to ingest streaming HL7 messages and planned to use Kafka with Smolder to achieve this. I'm having a little trouble understanding how Kafka can run on databricks. So far I've installed Kafka and started the cluster but the start command ju...

Data Engineering
Reply
  • 1300 Views
  • 3 replies
  • 5 kudos
View Replies
Latest Reply
Chris_Shehu
Valued Contributor III
  • 5 kudos

@Jose Gonzalez​  Sorry for the delay. I've actually already looked at the provided links. The first one is for AWS. I'm currently using Azure. The KAFKA setup is working..I think? I'm stuck at ok so I have a server listening now what? How do I send/r...

  • 5 kudos
2 More Replies
Mado
by • Valued Contributor II
  • 919 Views
  • 1 replies
  • 3 kudos

Resolved! How to specify directory when reading table?

Hi, Assume that I have created several tables in Databricks workspace. If two tables have similar names but stored in different folders, how I can read them by "spark.read.table". I ask this question because input to the "spark.read.table" is only ta...

Data Engineering
Reply
  • 919 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

spark.read.table reads table registered in catalog/metastore. If you have multiple databases in metastore, just prefix table name with database name db.table

  • 3 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors
View all