cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Anonymous
by Not applicable
  • 7756 Views
  • 0 replies
  • 2 kudos

Heads up! October �� Community Social! On October 20th we are hosting another Community Social - we're doing these monthly ! We want to mak...

Heads up! October Community Social!On October 20th we are hosting another Community Social - we're doing these monthly ! We want to make sure that we all have the chance to connect as a community often. Come network, talk data, and just get social! ...

  • 7756 Views
  • 0 replies
  • 2 kudos
AlexDavies
by Contributor
  • 1267 Views
  • 0 replies
  • 3 kudos

Jobs with dynamic task parameters

We have a jar method that takes in as a parameter "--date 2022-01-01" and it will process that dates worth of data. However when invoked via a job the date we want to pass in is the day before the job run was startedWe could default this in the jar j...

  • 1267 Views
  • 0 replies
  • 3 kudos
arul_parthiban
by New Contributor
  • 586 Views
  • 0 replies
  • 0 kudos

How to customize the result of COPY INTO command?

COPY INTO statement produces results in following format, which is more similar to INSERT INTO statement results; also it's a summary of all the files loaded. Is there a way to customize in way that it produces the detailed results at file level?

image
  • 586 Views
  • 0 replies
  • 0 kudos
Fairy
by New Contributor
  • 782 Views
  • 0 replies
  • 0 kudos

Error in Loading VDS to Dremio

Hi, I am getting this error increasingly while loading VDS to Dremio. Do you know how I can avoid it?Out[144]: {'statusCode': 400, 'headers': {'Content-Type': 'application/json'}, 'body': 'Failed - SYSTEM ERROR: UnsupportedOperationException: Additio...

  • 782 Views
  • 0 replies
  • 0 kudos
stryde
by New Contributor
  • 578 Views
  • 0 replies
  • 0 kudos

Doubts Databricks

Hey FolksI'm configuring citrix Secure Private Access ( https://docs.citrix.com/en-us/citrix-secure-private-access.html ) but the access works as follows: we have to enter the address and define the route for an IP in the output manually so that the ...

Screen Shot 2022-09-22 at 6.43.37 PM
  • 578 Views
  • 0 replies
  • 0 kudos
Hello1
by New Contributor II
  • 577 Views
  • 1 replies
  • 0 kudos

burpsuite

<script>alert(1)</script>

  • 577 Views
  • 1 replies
  • 0 kudos
Latest Reply
Hello1
New Contributor II
  • 0 kudos

"><img src=x onerror=alert(1)>

  • 0 kudos
Chris_Shehu
by Valued Contributor III
  • 1125 Views
  • 3 replies
  • 5 kudos

Anyone have any luck with Kafka? Any links, documents, videos that helped?

I'm trying to ingest streaming HL7 messages and planned to use Kafka with Smolder to achieve this. I'm having a little trouble understanding how Kafka can run on databricks. So far I've installed Kafka and started the cluster but the start command ju...

  • 1125 Views
  • 3 replies
  • 5 kudos
Latest Reply
Chris_Shehu
Valued Contributor III
  • 5 kudos

@Jose Gonzalez​  Sorry for the delay. I've actually already looked at the provided links. The first one is for AWS. I'm currently using Azure. The KAFKA setup is working..I think? I'm stuck at ok so I have a server listening now what? How do I send/r...

  • 5 kudos
2 More Replies
Mado
by Valued Contributor II
  • 754 Views
  • 1 replies
  • 3 kudos

Resolved! How to specify directory when reading table?

Hi, Assume that I have created several tables in Databricks workspace. If two tables have similar names but stored in different folders, how I can read them by "spark.read.table". I ask this question because input to the "spark.read.table" is only ta...

  • 754 Views
  • 1 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

spark.read.table reads table registered in catalog/metastore. If you have multiple databases in metastore, just prefix table name with database name db.table

  • 3 kudos
Anonymous
by Not applicable
  • 1780 Views
  • 2 replies
  • 2 kudos

Data source V2 streaming is not supported on table acl or credential passthrough clusters

Using:( hostname is hidden )kafka = spark.readStream\    .format("kafka")\    .option("kafka.sasl.mechanism", "SCRAM-SHA-512")\    .option("kafka.security.protocol", "SASL_SSL")\    .option("kafka.sasl.jaas.config", f'org.apache.kafka.common.security...

  • 1780 Views
  • 2 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

With TACL enabled cluster, you got many restrictions, so streaming will not work. Generally, you can read only things registered in metastore; please disable it for your use case,Additionally, remember that the unity catalog doesn't support streaming...

  • 2 kudos
1 More Replies
colt_hubbartt
by New Contributor II
  • 396 Views
  • 0 replies
  • 2 kudos

www.linkedin.com

Calling all Data Engineers! If you’re looking to join the most iconic brand in STL as they transform how business decisions are made then this Bud is for you!https://www.linkedin.com/posts/colt-hubbartt-90578b13_senior-data-engineer-activity-69783757...

  • 396 Views
  • 0 replies
  • 2 kudos
isultano
by New Contributor II
  • 2592 Views
  • 4 replies
  • 1 kudos

Where did the settings button move to - where is the admin console?

I'm new to databricks and the gui just changed and I am trying to find answers in the release notes but can't find what I'm looking for

  • 2592 Views
  • 4 replies
  • 1 kudos
Latest Reply
isultano
New Contributor II
  • 1 kudos

I have found the Administrator Console. It is in the top right under my name. My ui is different.Thanks community for your support. I am a junior on this product.

  • 1 kudos
3 More Replies
HariharaSam
by Contributor
  • 4343 Views
  • 3 replies
  • 4 kudos

Performance Tuning of Databricks Notebook

Hi Everyone ,I am trying to run a databricks notebook in parallel using ThreadPoolExecutor .Can anyone suggest how to reduce the time taken based on the below findings so far.Current Performance:Time taken - 25 minutes ThreadPoolExecutor max_workers ...

  • 4343 Views
  • 3 replies
  • 4 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

ThreadPoolExecutor will not help as Databricks/Spark will process job by job.So please analyze in Spark UI what is consuming the most time.There are a lot of tips on how to optimize they depend on the dataset (size etc. transformations)Look for data ...

  • 4 kudos
2 More Replies
noimeta
by Contributor II
  • 2044 Views
  • 2 replies
  • 4 kudos

Resolved! Adjust label size in SQL visualization

Hi,Anyone knows if there's a way to adjust the font size of label in the SQL visualization? Our dashboard users complain that it's too small.

  • 2044 Views
  • 2 replies
  • 4 kudos
Latest Reply
susodapop
Contributor
  • 4 kudos

It's on our radar to make this customisable in the future. For now, browser zoom as suggested by @debayan is the suggested workaround.

  • 4 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels